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MEMORANDUM 


FOR  UNDER  SECRETARY  OF  DEFENSE  FOR  ACQUISITION 
COMPTROLLER  OF  THE  DEPARTMENT  OF  DEFENSE 


SUBJECT;  Audit  Report  on  the  Review  of  the  All  Source  Analysis  System  as  a  Part  of 
the  Audit  of  the  Effectiveness  of  the  Defense  Acquisition  Board  Review 
Process— FY  1993  (Report  No.  93-087) 


We  are  providing  this  final  report  for  your  information  and  use.  Comments  on 
a  draft  of  this  report  were  considered  in  preparing  the  final  report.  DoD 
Directive  7650.3  requires  that  all  audit  recommendations  be  resolved  promptly. 
Therefore,  all  addressees  must  provide  final  comments  on  the  unresolved 
recommendations  by  June  21,  1993.  See  the  "Management  Comments"  section  at  the 
end  of  the  executive  summary  and  die  finding  for  the  unresolved  recommendations  and 
the  specific  requirements  for  your  comments. 

As  required  by  DoD  Directive  7650.3,  the  comments  must  indicate  concurrence 
or  nonconcurrence  in  the  finding  and  each  recommendation  addressed  to  you. 
Recommendations  are  subject  to  resolution  in  the  event  of  nonconcurrence  or  failure  to 
comment.  We  also  ask  that  your  comments  indicate  concurrence  or  nonconcurrence 
wiA  the  material  internal  control  weaknesses  highlighted  in  Part  I. 

We  appreciate  the  courtesies  extended  to  the  audit  staff.  If  you  have  questions 
on  this  report,  please  contact  Program  Director  Russell  A.  Rau  at  (703)  693-0186 
(DSN  223-0186).  Appendix  E  lists  the  planned  distribution  of  this  report. 


Edward  R.  Jones 

Deputy  Assistant  Inspector  General 
for  Auditing 


Enclosure 


cc: 

Secretary  of  the  Army 

Assistant  Secretary  of  Defense  for  Command,  Control, 
Communications,  and  Intelligence 


Office  of  the  Inspector  General,  DoD 


Audit  Report  No.  93-087  April  20,  1993 

Project  No.  2AE-0033.02 


REVIEW  OF  THE  ALL  SOURCE  ANALYSIS  SYSTEM  AS  A  PART 
OF  THE  AUDIT  OF  THE  EFFECTIVENESS  OF  THE  DEFENSE 
ACQUISITION  BOARD  REVIEW  PROCESS-FY  1993 


EXECUTIVE  SUMMARY 


Introduction.  The  All  Source  Analysis  System  (ASAS)  is  a  ground-based  mobUe 
intelligence-processing  system  designed  to  provide  automated  support  to  Army  combat 
commanders.  The  ASAS  will  provide  commanders  a  timely  and  comprehensive 
understanding  of  opposing  force  deployments,  capabilities,  and  potential  courses  of 
action.  The  ASAS  Program  is  an  evolutionary  development  effort  consisting  of 
five  blocks.  As  a  result  of  our  audit,  the  Under  Secretary  of  Defense  for  Acquisition 
(USD[A])  redesignated  the  Defense  Acquisition  Board  (DAB)  program  review 
scheduled  for  March  1993  as  a  Milestone  n  review  rescheduled  for  May  1993. 

Objectives.  The  overall  audit  objective  was  to  evaluate  the  DAB  review  process  for 
the  acquisition  of  the  ASAS.  Specifically,  we  assessed  the  adequacy  of  testing  to 
support  decisions  to  field  the  Block  I  system  and  award  the  contract  for  the  Block  U 
system.  We  also  reviewed  the  justification  for  not  conducting  a  DAB  Milestone  n 
review.  In  addition,  we  evaluated  the  scope  of  issues  defined  and  documentation 
required  for  the  March  1993  ASAS  Program  review.  Further,  we  assessed  compliance 
with  DoD  acquisition  policy. 

Audit  Results.  The  DAB  process  was  not  effective  for  the  ASAS  Program.  Testing 
conducted  and  planned  was  inadequate  to  support  a  decision  to  field  the  Block  I  system 
or  award  the  engineering  and  manufacturing  development  (EMD)  contract  for  the 
Block  n  system.  No  DAB  milestone  review  was  held,  and  none  was  planned  until 
1997,  although  the  Program  had  been  in  development  since  1983  and  had  cost  about 
$1.4  billion  through  FY  1992.  The  DAB  program  review  planned  for  March  1993  did 
not  include  the  materiel  release  decision  for  fiock  I  and  did  not  ensure  preparation  of 
jdl  documentation  needed  for  a  thorough  review. 

Internal  Controls.  The  audit  identified  material  internal  control  weaknesses.  Controls 
were  not  effective  for  the  evolutionary  acquisition  strategy  and  not  implemented  for  the 
ASAS  Program  when  it  entered  major  DAB  program  status.  Part  I  discusses  these 
internal  control  weaknesses. 

Potential  Benefits  of  Audit.  Potential  monetary  benefits  are  not  readily  quantifiable 
because  the  cost  of  correcting  Block  I  deficiencies  after  fielding  is  unknown 
(Appendix  C).  Savings  would  result  from  a  proper  assessment  of  the  ASAS  Program 
before  fielding  the  Block  I  system  or  awarding  the  EMD  contract  for  the  Block  II 
system. 

Summary  of  Recommendations.  We  recommended  that  a  DAB  Milestone  II  review 
of  the  ASAS  Program  be  conducted  and  that  remaining  FY  1993  ASAS  funding  be 
withheld  until  an  acquisition  decision  memorandum  is  issued  following  the  Milestone  U 
review. 


Management  Comments.  The  USD(A)  partially  concurred  with  our  finding  and 
recommendations  and  the  Comptroller  of  Ae  Department  of  Defense  concurred.  A  full 
discussion  of  their  responses  is  in  Part  II;  the  complete  text  of  their  comments  is  in 
Appendix  B  and  Part  IV.  We  request  that  the  USD(A)  reconsider  his  position  and  that 
both  he  and  the  Comptroller  provide  additional  comments  to  the  report  by  June  21, 
1993. 
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Introduction 


Background 


The  All  Source  Analysis  System  (ASAS)  is  a  ground-based  mobile  intelligence 
processing  system  designed  to  provide  automated  support  to  Army  combat 
commanders.  The  ASAS  will  provide  commanders  a  timely  and  comprehensive 
understanding  of  opposing  force  deployments,  capabilities,  and  potential  courses 
of  action.  The  ASAS  Program  is  an  evolutionary  development  effort  consisting 
of  five  increments  (or  blocks).  As  of  June  1992,  the  cost  to  acquire  the  ASAS 
through  Block  III  was  estimated  at  $5.89  billion;  $1.60  billion  for  Block  I  and 
$4.29  billion  for  Blocks  II  and  m.^ 

The  Block  I  portion  of  the  ASAS  Program  provides  an  initial  capability  to 
perform  four  of  six  primary  functions;  intelligence  collection  management, 
signal  intelligence  management,  situation  analysis,  and  target  analysis.  The 
ASAS  Block  I  program  has  been  in  development  since  1983.  Through 
FY  1992,  the  Army  has  expended  $1,036  billion  for  development  and 
$331  million  for  production  of  12  Block  I  systems  to  be  deployed  from  FY  1993 
through  FY  1995  to  11  priority  (Force  Package  1)  units  plus  the  Army 
Intelligence  School.  Block  I  development  was  not  performed  under  Defense 
Acquisition  Board  (DAB)  cognizance.  Rather,  the  Joint  Oversight  Group  had 
responsibility  for  the  Block  I  program. 

Beginning  in  FY  1993,  the  Block  n  program  will  replace  the  ASAS  Block  I 
system  with  Army  Tactical  Command  and  Control  System  (ATCCS)  common 
hardware  and  software  but  will  not  increase  the  number  of  primary  functions 
beyond  the  four  functions  already  attained.  The  Block  HI  upgrade,  scheduled 
for  FYs  1998  to  2003,  will  develop  the  final  two  functions;  electronic  warfee 
support  and  operations  security  support.  The  Blocks  IV  and  V  upgrades  during 
FYs  2002  to  2012  will  improve  and  enhance  existing  capabilities.  Block  V  is 
intended  to  fully  satisfy  all  user  requirements  as  stated  in  the  ASAS  Required 
Operational  Capability  (ROC). 

Blocks  IV  and  V  are  not  included  in  the  ASAS  acquisition  program  baseline. 
According  to  the  ASAS  Program  Manager,  the  cost  of  Blocks  IV  and  V  has  not 
been  determined. 


^Source:  Program  Deviation  Report,  June 23,  1992.  All  Program  costs  and 

fimding  became  unclassified  on  December  1,  1992.  All  dollars  mentioned  in  this  report 

» are  current  (then-year)  dollars.  ,  o  »  t 

^  A  committee  of  Army  and  Air  Force  general  officers  and  Office  of  the  Secretary  or 
Defense  (OSD)  officials  that  directed  the  ASAS  Program  from  1982  to  1987. 
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Introduction 


Objectives 


The  overall  audit  objective  was  to  evaluate  the  DAB  review  process  for  the 
acquisition  of  the  ASAS,  Specifically,  we  assessed  the  adequacy  of  testing  to 
support  decisions  to  field  the  Block  I  system  and  award  the  contract  for  the 
Block  II  system.  We  also  reviewed  the  justification  for  not  inducting  a  DAB 
Milestone  II  review.  In  addition,  we  evaluated  the  scope  of  issues  defined  and 
documentation  required  in  support  of  the  scheduled  March  1993  DAB  program 
review.  Further,  we  assessed  compliance  with  DoD  Directive  5000. 1 ,  "Defense 
Acquisition,"  February  23,  1991;  DoD  Instruction  5000.2,  "Defense 

Acquisition  Management  Policies  and  Procedures,"  February  23,  1991;  and 
DoD  Manual  5000.2-M,  "Defense  Acquisition  Management  Documentation  and 
Reports,"  February  23,  1991.  Additionally,  we  reviewed  applicable  internal 
controls. 


Scope 


We  performed  this  program  audit  from  July  through  December  1992  and 
reviewed  records  dated  from  1982  through  1992  relating  to  the  ASAS.  We 
performed  this  audit  in  accordance  with  auditing  standards  issued  by  the 
Comptroller  General  of  the  United  States,  as  implemented  by  the  Inspector 
General,  DoD,  and  accordingly  included  such  tests  of  internal  controls  as  were 
deemed  necessary.  We  discussed  issues  related  to  the  ASAS  and  the  DAB 
review  process  with  Office  of  the  Secretary  of  Defense  (OSD),  Defense 
Intelligence  Agency,  Joint  Staff,  and  Army  personnel  responsible  for  the 
preparation  or  review  of  DAB-required  documents.  Appendix  D  lists  activities 
visited  or  contacted. 


Internal  Controls 


The  audit  identified  material  internal  control  weaknesses  as  defined  by  Public 
Law  97-255,  Office  of  Management  and  Budget  Circular  A-123,  and  DoD 
Directive  5010.38.  Controls  were  not  effective  for  the  review  and  approval  of 
the  ASAS  evolutionary  acquisition  strategy.  Additionally,  controls  were  not 
implemented  on  the  ASAS  Program  when  it  entered  major  DAB  program  status 
to  ensure  preparation  of  an  acquisition  pro^m  baseline  and  other  required 
documentation.  Recommendations  1.  and  2.,  if  fully  implemented,  will  corr^t 
these  weaknesses  for  the  ASAS  Program.  Monet^  benefits  are  not  r^dily 
quantifiable  because  the  cost  of  correcting  Block  I  deficiencies  after  fielding  is 
unknown  (Appendix  C).  Copies  of  the  final  report  will  be  provided  to  the 
senior  officials  responsible  for  internal  controls  within  OSD  and  the  Department 
of  the  Army. 
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Introduction 


Prior  Audits  and  Other  Reviews 


During  the  last  5  years,  there  has  been  no  audit  coverage  on  the  ASAS  relating 
to  the  DAB  review  process. 


Other  Matters  of  Interest 


On  Septembers  and  11,  1992,  we  met  with  the  Deputy  Director,  Intelligence 
Programs  Support  Group  (Office  of  the  Assistpt  Secretarj^  of  Defense  for 
Command,  Control,  Communications  and  Intelligence)  to  discuss  issues  and 
documentation  for  the  program  review,  then  scheduled  for  January  1993.  He 
request^  our  assistance  in  defining  requirements  for  the  review.  Based  on 
current  results  of  our  audit,  we  stated  that  a  Milestone  II  review  rather  than  a 
program  review  may  be  needed.  At  a  minimum,  we  considered  identification  of 
a  Block  II  low-rate  initial  production  (LRIP)  quantity  and  submission  of  an 
Operational  Requirements  Document  (ORD)  to  be  critical.  The  working  group 
of  Ae  Command,  Control,  Communications  and  Intelligence  (C^I)  Systems 
Committee  held  planning  meetings  on  September  24  and  October  1,  1992.  On 
October  9,  1992,  the  Committee  Chair  issued  a  Committee  Memorandum  which 
provided  guidance  for  the  program  review,  then  planned  for  March  1993.  With 
these  meetings  and  the  memorandum,  the  Committee  took  positive  steps  to 
control  the  ASAS  Program  by  defining  issues  to  be  addressed  and  requiring 
preparation  of  key  documentation  for  the  program  review. 
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Part  n  -  Finding  and  Reconunendations 


5 


Effectiveness  of  the  Defense  Acquisition 
Board  Review  Process 

The  DAB  review  process  was  not  effective  for  the  ASAS  Program. 
Specifically,  testing  and  review  prerequisites  did  not  adequately  support 
a  decision  to  field  the  Block  I  system  or  award  the  engineering  and 
manufacturing  development  (EMD)  contract  for  the  Block  II  ^stem.  No 
DAB  milestone  review  was  held,  and  none  was  planned  until  the 
Milestone  III,  Production  and  Deployment,  decision  for  the  Block  n 
system  in  1997,  although  the  Program  had  been  in  development  since 
1983  and  had  cost  $1.39  billion^  in  Development  and  Procurement  funds 
through  FY  1992.  The  DAB  program  review  planned  for  March  1993 
did  not  include  the  materiel  release  decision  for  Block  I  and  did  not 
require  preparation  of  all  documentation  needed  for  entry  of  a  major 
Defense  acquisition  program  into  EMD.  As  a  result,  the  ASAS  Program 
had  not  had  the  necessary  direction,  and  the  planned  Dj^  review  lacked 
essential  information  upon  which  to  decide  to  proceed  with  the  Program. 


Background 


The  ASAS  is  a  major  Defense  acquisition  program  (Acquisition  Category  I D) 
for  which  the  Under  Secretary  of  Defense  for  Acquisition  (USD[A])  is  the 
milestone  decision  authority.  The  USD(A)  chairs  the  DAB  for  milestone 
reviews,  and  the  DAB  is  supported  by  three  committees.  The  committee 
responsible  for  oversight  of  the  ASAS  Program  is  the  C^I  Systems  Committee 
chair^  by  the  Deputy  Assistant  Secretary  for  Defense  (Strategic  and  Tactical 
Command,  Control  and  Communications),  Office  of  the  Assistant  Secretary  of 
Defense  for  C^I. 

Reviews.  The  DAB  process  provides  for  four  types  of  reviews: 
milestone,  special  program,  milestone  committee,  and  non-milestone  committee 
reviews. 


Milestone  Reviews.  The  purpose  of  a  milestone  review  is  to 
determine  where  the  program  is  versus  where  it  should  be,  where  the  program  is 
going  and  how  the  program  manager  proposes  to  get  there,  what  risks  exist  in 
the  program  and  how  the  program  manager  will  identify  and  close  those  risks, 
and  if  the  program  manager's  proposed  approach  is  affordable.  Documentation 
is  the  primary  means  for  the  functional  staff  ^d  the  program  manager  to 
provide  the  milestone  decision  authority  with  the  information  needed  to  inake  a 
milestone  decision.  The  milestone  decision  authority  documents  completion  of 
the  milestone  review  by  issuing  an  acquisition  decision  memorandum  (ADM). 


^  $1,368  billion  for  Block  I  and  $22  million  for  Block  H;  $1,059  billion  for  development  and 
$331  million  for  procurement. 
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Effectiveness  of  the  Defense  Acquisition  Board  Review  Process 


The  ADM  for  Milestone  II  should  approve  entry  into  Phase  II,  approve  Ae 
proposed  or  modified  acquisition  strategy  and  development  baseline,  establish 
exit  criteria  that  must  be  accomplish^  during  Phase  n,  and  identify  LRIP 
quantities,  if  appropriate. 

Special  Program  Reviews.  The  USD(A)  may  hold  sp^ial 
program  reviews  between  milestone  reviews.  Agenda  topics  shall  be  identified-  - 
at  least  30  calendar  days  before  the  scheduled  review.  Documentation  required 
shall  be  tailored  to  the  specific  requirements  for  the  program  review  but  shall 
not  exceed  the  requirements  for  a  milestone  review  without  specific 
authorization  of  the  USD(A). 

Milestone  Committee  Reviews.  The  purposes  of  the  DAB 
committee  reviews  are  to  verify  that  exit  criteria  and  the  minimum  required 
accomplishments  of  the  phase  preceding  the  milestone  have  been  completed; 
provide  an  independent  assessment  of  the  program  which,  together  with  the 
Component’s  Integrated  Program  Summary,  is  the  basis  for  the  DAB  review; 
and  make  recommendations  on  trade-offs  among  cost,  schedule,  and 
performance  proposed  by  the  program  manager  for  decision  by  the  USD(A). 

Non-Milestone  Committee  Reviews.  The  thr^  Committees 
convene  periodically  for  special  reviews,  apart  from  the  DAB  milestone  review 
process,  as  approved  by  the  USD(A),  In  general,  the  procedures  for  milestone 
reviews  apply.  However,  specific  requirements  are  t^ored  to  meet  schedule 
constraints  or  special  review  considerations.  As  a  minimum,  a  planning 
meeting  will  discuss  plans  and  set  requirements  for  the  Committee  review. 
Within  a  week  of  this  meeting,  the  Committee  Chair  will  issue  a  Committee 
Memorandum  which  will  state  clearly  the  purpose  of  the  special  review; 
establish  the  timeline  of  events;  identify  the  documentation  required;  and 
describe  the  review  issues,  agenda,  and  responsibilities. 

Test  and  Evaluation  Programs.  DoD  Instruction  5000.2  establishes 
policies  for  test  and  evaluation  programs,  stating  the  program  should  be 
structured  to  provide: 

o  Essential  information  for  assessment  of  acquisition  risk  and  for 
decisionmaking, 

o  Verification  of  attainment  of  technical  performance  specifications  and 
objectives, 

o  Verification  that  systems  are  operationally  effective  and  suitable  for 
intended  use,  and 

o  Essential  information  in  support  of  decisionmaking. 

The  Test  and  Evaluation  Master  Plan  (TEMP)  documents  the  approved  overall 
test  and  evaluation  program  for  a  system.  Supporting  developmental  and 
operational  test  plans,  such  as  a  Test  and  Evaluation  Plan  (TEP),  are  generated 
to  provide  the  details  and  schedule  of  testing. 
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Effectiveness  of  the  Defense  Acquisition  Board  Review  Process 


At  critical  program  decision  points,  test  results  are  documented  to  supp^tj^e 
decision  by  the  milestone  decision  authority.  At  the  Milestone  II  (EMD) 
decision,  a  Developmental  Test  and  Evaluation  Report  is  required  to  provide  the 
results  of  developmental  testing.  Additionally,  an  Early  Operational  Assessment 
Report  is  required  to  support  an  LRIP  decision  at  Milestone  n.  Operational 
assessments  at  the  Milestone  II  decision  point  can  be  based  on  computer 
modeUng,  simulation,  or  analysis  of  system  requirements  ^d  design 
specifications,  where  production  representative  units  are  unavailable  for  testing. 

At  the  Milestone  HI,  Production  and  Deployment,  decision,  the  Cornponem 
requires  the  Operational  Test  and  Evaluation  Report  and  the  Beyond  LRIP 
Report  from  the  Director,  Operational  Test  and  Evaluation.  These  reports 
assess  the  initial  operational  test  and  evaluation  perform^  and  provi^ 
conclusions  concerning  operational  effectiveness  and  suitability.  The  DoD 
Component  must  certify  that  a  system  is  ready  for  initial  operational  test  ^d 
evaluation  (IOT«feE)  and  obtain  approval  of  the  TEP  before  starting  lOT&E. 
Operational  testing  must  be  performed  on  production  or  production- 
representative  articles  operated  by  typical  users  under  combat  conditions  against 
threat  representative  forces. 


Adequacy  of  Testing  and  Reviews 


One  minimum  required  accomplishment  for  EMD  is  that  test  results  accurately 
show  performance  under  operational  conditions.  The  ASAS  Block  I  Program 
has  been  in  EMD  since  1983.  The  Block  11  program  was  scheduled  to  start 
EMD  in  March  1993.  However,  testing  conducted  and  testing  planned  were 
inadequate  to  support  a  decision  to  field  Block  I  equipment  or  award  the 
Block  n  EMD  contract.  Additionally,  Armv  use  of  an  accelerated  acquisition 
strategy  called  "limited  procurement-urgent""*  circumvented  established  design 
and  production  readiness  review  and  configuration  audit  requirements  for 
Block  I.  Therefore,  thorough  operational  test  and  evaluation  of  Block  I  before 
fielding  is  essential. 

Block  I  Testing.  The  Block  I  test  and  evaluation  program  did  not 
support  the  production  decision  in  1989  for  the  first  units  moduced. 
Additionally,  the  test  program  had  not  provided  for  full  evaluation  of  product 
qualification  or  operational  suitability  and  effectiveness  before  fielding  the 
Block  I  ASAS.  In  our  opinion,  the  test  program  provided  an  appearance  of 
successful  progress  rather  than  complete  information  upon  which  to  base 
Program  decisions. 

Testing  to  Date.  The  Force  Development  Test  and  Experimentation 
(FDT&E),  a  combined  developmental  and  operational  test  conducted  in 
November  and  December  1989  by  Test  and  Evaluation  Command,  _was 
discontinued  due  to  problems  with  the  message  parser,  a  unit  that  identifies 


^  The  "limited  procurement-urgent"  strategy  does  not  comply  with  current  acquisition  directives 
and  instructions  for  major  Defense  acquisition  programs. 
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incoming  messages  and  routes  them  to  the  appropriate  database.  The  purpose  of 
the  test,  according  to  the  FDT&E  test  report  of  February  1990,  was  to  establish 
a  performance  baseline  to  support  the  Operational  Test  and  Evaluation  Agency's 
continuous  evaluation  of  AS  AS.  Due  to  the  problem  with  the  parser,  FDT&E 
fail^  to  establish  the  required  performance  baseline  for  lOT&E.  Rather  than 
correcting  the  parser  deficiencies  and  reaccomplishing  FDT&E,  the  Army 
proceeded  wiA  more  advanced  testing. 

The  Preproduction  Qualification  Test  (PPQT)  is  a  series  of  technical  tests  that 
evaluates  the  design  of  a  system  relative  to  the  performance  requirements  over 
the  specified  operational  and  environmental  range  before  design  release  for 
production.  The  focus  of  PPQT  is  the  proposed  design,  not  the  production  end 
item's  meeting  contract  technical  performance  specifications  that  is  the  focus  of 
production  verification  testing.  The  Army  Test  and  Evaluation  Command 
conducted  a  PPQT  from  January  through  July  1992  to  determine  the  materiel 
suitability  of  ASAS  Block  I  units  already  produced  for  release  to  priority  units. 
These  tests  were  limited  compared  to  formal  Production  Qualification  testing, 
which  is  required  by  Army  regulations  to  support  a  materiel  release  decision, 
requiring  demonstration  versus  fiiU  test  and  evaluation  of  factors  such  as 
maintainability  and  reliability.  Although  a  formal  report  was  not  yet  completed, 
the  Army  Materiel  Systems  Analysis  Activity  rated  ASAS  as  having  low 
technical  risk  based  on  performance,  supportability,  and  reliability  and  stated 
"We  see  no  reason  not  to  proceed  on  and  conduct  the  lOT&E." 

In  our  opinion,  PPQT  is  not  intended  to  support  a  materiel  release  decision  or 
certification  of  readiness  for  lOT&E.  The  certification  should  confirm  that  the 
test  article  is  a  production  or  production-representative  unit  as  determined  by 
configuration  audits  and  the  full  range  of  technical  testing.  On  September  2, 
1992,  the  Program  Manager  certified  the  Block  I  system  was  ready  to  enter 
lOT&E. 

An  lOT&E  is  required  before  entry  into  full-rate  production,  using  production 
or  production-representative  articles,  typical  users,  and  realistic  operating 
conditions.  The  lOT&E  was  conducted  from  Septembers,  1992,  through 
October  11,  1992,  to  assess  the  operational  suitability  and  effectiveness  of  the 
ASAS  Block  I  system  and  to  support  a  materiel  release  decision  in  FY  1993. 
The  lOT&E  was  based  upon  the  negative  assurance  of  the  Army  Materiel 
Systems  Analysis  Activity  on  the  preliminary  results  of  PPQT  and  without 
necessary  support  for  the  certification  to  proceed  with  lOT&E.  As  a  result,  we 
consider  the  entry  into  lOT&E  to  have  been  unduly  expedited. 

Initial  Operational  Test  and  Evaluation  Invalid.  Block  I  lOT&E  did 
not  include  testing  of  interoperability  as  required  by  law,  was  not  conducted  on 
a  production  representative  system,  and  was  scheduled  after  the  production  of 
most  ASAS  Block  I  units.  Further,  the  Test  and  Evaluation  Plan  was  not  timely 
and  did  not  provide  for  determination  of  operational  effectiveness  and 
suitability. 
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Testing  of  Interoperability.  Testing  of  interoperability  for  each 
ATCCS  component  system  during  lOT&E  is  required  by  Congress. 
Specifically,  the  FY 1991  Defense  Appropriations  Conference  Report, 
October  24,  1990,  stated: 

The  conferees,  therefore,  direct  the  Army  to  revise  its  test 
and  evaluation  master  plan  for  each  ATCCS  component 
system  to  include  total  system  interoperability  testing  at  each 
development  and  operational  testing  milestone. 

The  Collateral  Enclave,  a  critical  component  of  the  ASAS  that  is  the  interface  to 
and  that  provides  interoperability  with  the  ATCCS,  was  not  tested  during 
lOT&E.  Although  the  TEMP  included  the  requirement  for  interoperability 
testing,  it  excluded  interoperability  testing  from  IOT«&E.  Therefore, 
interoperability  testing  was  not  included  in  the  TEP  and  was  not  conducted.  In 
August  1992,  the  Program  Office  informed  us  that  the  Collateral  Enclave  was 
scheduled  to  be  tested  during  Phase  I  of  the  ATCCS  Early  User  Test  and 
Experimentation  (EUT&E)  October  26  to  30,  1992.  Then,  on  October  15, 
1992  (4  days  after  lOT&E  ended),  the  Program  Office  informed  us  that  the 
Collateral  Enclave  would  be  demonstrated  during  Phase  I  and  tested  during 
Phase  II  of  EUT&E  in  May  1993. 

Production  System  and  Decision.  lOT&E  must  be  conducted 
on  production  or  production-representative  systems  before  the  full-rate 
production  decision.  DoD  Instruction  5000.2  defines  lOT&E  as: 

All  operational  test  and  evaluation  conducted  on  production 
or  production  representative  articles,  to  support  the  decision 
to  proceed  beyond  low-rate  initial  production.  It  is 
conducted  to  provide  a  valid  estimate  of  expected  system 
operational  effectiveness  and  operational  suitability. 


The  ASAS  Block  I  lOT&E  was  performed  on  an  interim  system  developed 
under  unique  Army  "limited  procurement-urgent"  procedures,  not  a  production 
system.  Specifically,  the  Block  I  configuration  is  not  intended  to  meet  the 
operational  requirements  in  the  approv^  ROC  of  August  1991.  Required 
reviews  and  other  controls  that  normally  support  lOT&E  prior  to  a  production 
decision  were  bypassed  at  the  time  because  of  the  expedited  Army  procedures, 
including  production  readiness  reviews  and  functional  and  physical 
configuration  audits.  Further,  Block  I  lOT&E  did  not  support  a  production 
decision  as  most  of  the  Block  I  equipment,  planned  only  for  the  Army's  priority 
units,  had  already  been  procured. 

Test  and  Evaluation  Plan.  DoD  Instruction  5000.2  requires  the  TEP  to  be 
provided  to  the  Director,  Operational  Test  and  Evaluation  (DOT&E),  at  least 
60  days  before  start  of  lOT&E,  and  the  DOT&E  had  so  advised  the  Under 
Secretary  of  the  Army  (Operations  Research)  in  a  memorandum  dated 
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March  30,  1992.  However,  the  TEP  was  received  for  review  by  the  DOT&E 
on  August  21,  1992,  only  18  days  before  lOT&E  was  to  begin.  On 
September  4,  1992,  the  DOT&E  stated  in  a  memorandum  to  the  Under 
Secretary  of  the  Army  (Operations  Research)  that  the  TEP  was  adequate  to 
support  a  baseline  effort  and  an  Army  materiel  release  decision  but  noted  that 
there  were: 


significant  limitations  .  .  .  that  would  prevent  both  a 
determination  of  operational  effectiveness  and  suitability  in 
accordance  with  the  OT&E  charter  and  the  subsequent 
recommendation  to  the  DAB  in  support  of  a  traditional 
Milestone  III  decision. 

The  memorandum  did  not  elaborate  on  the  significant  limitations.  However,  a 
tab  to  the  staff  package  supporting  the  memorandum  listed  the  following 
limitations:^ 

o  The  lOT&E  is  conducted  using  a  Heavy  Division.  Material  release 
decision  is  to  field  to  Corps,  Heavy  Divisions  and  Light  Divisions. 

o  The  ATCCS  interoperability  testing  is  demonstration  only  and  not 
included  as  a  part  of  the  formal  operational  testing. 

o  The  lOT&E  will  test  one  software  version;  fielding  wiU  be  on  another 
software  version. 

o  The  majority  of  required  interfaces  will  not  be  operationally  tested. 

o  The  majority  of  the  thresholds  used  by  the  Army  to  determine 
operational  effectiveness  and  suitability  are  opinion  polls. 

o  Much  of  the  data  gathered  to  support  the  lOT&E  will  be  in  a  garrison 
operation  rather  than  field  operations. 

Testing  Program  Invalid.  DoD  Instruction  5000.2  states  that 

Operational  test  and  evaluation  programs  shall  be  structured 
to  determine  the  operational  effectiveness  and  suitability  of  a 
system  under  realistic  combat  conditions  and  to  determine  if 
the  minimum  acceptable  operational  performance 
requirements  as  specified  in  the  Operational  Requirements 
Document  have  been  satisfied.  .  .  .  The  Test  and  Evaluation 
Master  Plan  will  be  used  to  generate  detailed  test  and 
evaluation  plans. 


The  tab  was  based  on  an  Institute  for  Defense  Analysis  Review  of  the  ASAS  TEP,  dated 
August  28,  1992. 


11 


Effectiveness  of  the  Defense  Acquisition  Board  Review  Process 


The  Instruction  also  states  that  all  hardware  and  software  changes  that  materially 
change  system  performance  should  be  adequately  tested  and  evaluated.  This 
was  not  the  case  with  the  use  of  the  Hawkeye  system  (an  OSD-sponsored 
balanced  technology  initiative  that  became  part  of  ASAS)  and  Collateral 
Enclave. 

The  ASAS  operational  testing  program  is  invalid.  Even  though  the  second 
operational  test,  the  FDT&E,  failed  to  establish  a  performance  baseline  for 
lOT&E,  the  testing  program  continued  as  if  there  had  been  no  problem. 
Further,  two  major  changes  occurred  to  the  Block  I  system  after  FDT&E  that 
were  not  properly  tested  prior  to  lOT&E.  First,  the  integration  of  the  Hawkeye 
system  into  the  ASAS  during  1991  was  a  major  change  to  the  configuration  that 
was  tested  during  FDT&E.  Then,  during  1992,  specifying  the  use  of  common 
hardware  and  software  in  the  Collateral  Enclave  (which  was  not  subsequently 
tested  during  lOT&E)  represented  another  significant  change  to  the  original 
system. 

The  ASAS  testing  program  does  not  comply  with  DoD  Instruction  5000.2.  The 
TEP  for  the  Block  I  lOT&E  does  not  conform  to  the  requirements  of  the  TOMP 
because  lOT&E  did  not  test  required  interfaces,  changed  critical  quantitative 
thresholds  to  non-critical  criteria,  and  reduced  the  test  from  45  to  26  days.  In 
our  opinion,  the  ASAS  operational  test  and  evaluation  program  is  inad^uate  to 
assess  operational  suitability  and  effectiveness  of  the  Block  I  system,  justify  a 
materiel  release  decision  for  the  Block  I  system,  or  support  award  of  the  EMD 
contract  for  the  Block  II  system.  In  particular,  ^  the  change  in  critic^ 
performance  thresholds  relegates  lOT&E  to  the  subjective  perceptions  of  the 
testers  instead  of  objective  performance  measures. 

DoD  Instruction  5000.2  defines  an  operational  assessment  as: 


An  evaluation  of  operational  effectiveness  and  operational 
suitability  made  ...  on  other  than  production  systems.  .  .  . 

Operational  assessments  may  be  made  at  any  time  .  .  .  but 
will  not  substitute  for  the  independent  operational  test  and 
evaluation  necessary  to  support  full  production  decisions. 

We  consider  the  testing  planned  for  Block  I  lOT&E  to  resemble  more  closely  an 
operational  assessment  than  an  operational  test  and  evaluation.  The  term 
"lOT&E",  as  applied  to  the  evaluation  made  of  the  ASAS,  gives  the  inaccurate 
impression  that  the  testing  performed  meets  the  rigorous  requirements  for 
operation^  test  and  evaluation. 

Impact  of  Inadequate  Testing.  As  of  September  30,  1992,  the  ASAS 
Program  had  cost  $1.39  billion  and  was  estimated  to  cost  another  $4.5  billion 
through  Block  IH  (FY  2003).  The  FY  1993  Defense  Appropriations  Act 
provided  $47.7  million  in  procurement  funds  to  begin  fielding  Block  I  and 
$57.1  million  in  RDT&E  funds  for  "Evolutionary  Acquisition."”  The 
inadequate  testing  of  the  ASAS  could  cause  the  Army  to  prematurely  decide  to 


Of  the  $57.1  million,  the  program  office  has  earmarked  $17.4  million  for  Block  I  and 
$39.7  million  for  Block  II. 
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field  Block  I  and  could  result  in  additional  costs  to  correct  problems  found  after 
fielding.  Reliance  on  test  results  based  on  inadequate  testing  could  also  impact 
the  direction  provided  by  the  DAB  on  the  ASAS  Block  H  program  entering 
EMD. 

In  our  opinion,  the  Army  should  not  have  used  expedited  "limited  procurement- 
urgent"  procedures  to  bypass  essential  reviews  in  Ihe  acquisition  process.  These  - 
expedited  procedures  are  not  suitable  for  development  or  test  and  evaluation 
programs. 


No  DAB  Milestone  n  Review 


No  DAB  Milestone  II  review  was  held  or  planned,  although  the  Army  considers 
ASAS  Block  I  to  be  essentially  complete  and  is  preparing  to  field  Block  I  and 
award  the  EMD  contract  for  Block  II.  In  our  opinion,  the  upcoming  DAB 
review  should  be  a  formal  Milestone  n  review  tiiat  results  in  the  rigorous 
oversight  provided  to  DAB  programs  and  should  be  conducted  before  the 
Block  II  EMD  contract  award. 

Reviews  Have  Been  Inadequate.  The  ASAS  has  been  a  DAB  program 
since  1987  but  has  never  had  a  DAB  milestone  review.  From  1982  to  1987,  the 
Program  was  under  the  purview  of  the  Joint  Oversight  Group,  which  in  1984 
declared  that  ASAS  Block  I  had  passed  Milestone  II  in  1983.  From  1988 
through  1990,  the  C^I  Systems  Committee  of  the  DAB  was  briefed  annually  on 
the  status  of  the  Program,  but  no  DAB  review  was  held  and  no  acquisition 
program  baseline  was  required  until  1991. 

Before  the  ASAS  Program  was  restructured  in  early  1991,  a  DAB  Milestone  HI 
review  for  Block  I  production  had  been  scheduled  for  July  1992.  In  November 
1991,  the  Army  submitted  the  Program's  first  acquisition  program  baseline, 
which  showed  a  DAB  program  review  in  April  1993,  just  prior  to  the  Block  I 
materiel  release  decision.  However,  this  program  review  was  scheduled 
9  months  after  the  planned  July  1992  award  of  the  Block  11  EMD  coni^act. 
Further,  the  DAB  Milestone  III  review,  now  for  the  purpose  of  deciding 
Block  II  production  and  development,  was  postponed  until  November  1997. 
These  events  reflect  the  Army  decision  to  cancel  production  of  systems 
developed  in  ASAS  Block  I  except  for  the  provision  of  a  limited  near-term 
capability  to  priority  units.  In  our  opinion,  these  events  also  reflect  a  desire  to 
postpone  a  DAB  milestone  review  indefinitely  and  to  award  the  Block  n  EMD 
contract  well  before  the  requested  date  for  a  DAB  program  review. 

In  December  1991,  the  USD(A)  approved  the  program  baseline.  However,  the 
USD(A)  scheduled  a  DAB  program  review  of  the  Block  II  program  for  July 
1992  and  stated  he  did  not  plan  to  reconvene  the  DAB  at  the  Block  I  materiel 
release  decision  point  "unless  currently  unforeseen  problems  arise."  We 
consider  a  DAB  review  to  be  required  before  the  Block  I  materiel  release 
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because  of  the  impact  of  Block  I  on  the  subsequent  Block  D  program.  Fielding 
the  Block  I  system,  especially  the  software,  before  full  operational  test  and 
evaluation  of  the  system  (including  all  interfaces)  can  increase  program  risk  on 
the  Block  I  and  II  programs. 

In  addition  to  the  baseline,  other  key  program  docunientation  required  by 
acquisition  regulations  for  a  major  Defense  acquisition  program  beyond 
Milestone  I  (Demonstration  and  Validation),  such  as  the  Acquisition  Plan, 
Acquisition  Strategy  Report,  System  Threat  Assessment  Report,  and  TEMP, 
were  not  required  by  or  submitted  to  OSD  until  1991.  Further,  an  Operational 
Requirements  Document,  begun  in  1991  and  not  scheduled  to  be  completed 
until  1995,  was  not  required  by  OSD  until  October  1992.  The  C^I  Committee 
Chair  has  requested  that  certain  DAB  documentation  be  prepared;  however,  the 
full  requirements  specified  for  a  Milestone  II  review  have  not  been  imposed. 

Procurement  of  Equipment  Without  a  Production  Decision.  Most 
equipment  needed  for  Block  I  materiel  release  has  been  procured.  The  plan  for 
Block  n  is  to  replace  all  Block  I  equipment  with  the  Block  II  equipment  before 
the  DAB  Milestone  III.  Therefore,  substantial  procurements  will  occur  without 
a  milestone  review  or  approval. 

Block  I  Procurements.  In  1986,  the  Army  directed  the 
procurement  of  a  limited  capability  configuration  of  ASAS.  In  1987,  the  Joint 
Oversight  Group  approved  acquisition  of  the  limited  capability  configuration 
systems  for  Ft.  Hood,  Texas,  and  in  1988  expanded  the  acquisition  to  include 
Europe.  In  January  1990,  the  Army  directed  restructuring  of  the  Program  to 
field  the  system  as  soon  as  possible  with  the  minimum  functionality  acceptable 
to  the  user.  Key  elements  in  the  restructuring  were  transition  to  common 
hardware  and  software  and  increased  procurement  funding  through  FY  2007. 

The  1991  restructure,  by  incorporating  the  Hawkeye  technology  into  the  ASAS 
Program,  doubled  the  systems  on  hand  from  6  to  12  and  eliminated  the  need  for 
further  procurement  of  Block  I  equipment.  In  March  1991,  due  to  the  success 
of  the  Hawkeye  system  during  Desert  Storm,  the  ASAS  Program  Manager 
included  Hawkeye  technology  in  the  acquisition  plan.  This  redirection  was 
briefed  to  the  cH  Systems  Committee  in  June  1991  and  was  approved  in 
December  1991.  Because  of  the  redirection,  the  limited  capability  configuration 
systems  and  Hawkeye  hardware  equipped  all  Army  Force  Package  1  units 
(three  Corps  and  eight  Divisions)  with  the  Block  I  limited  capability  version  of 
ASAS  for  fielding  from  FY  1993  through  FY  1995.  Full  production  was 
delayed  until  Block  n.  Thus,  a  Milestone  III  review  for  Block  I  production  was 
no  longer  relevant. 

Block  n  Plan.  As  of  August  1992,  the  Program  Manager 
planned  to  field  Block  II  in  "capability  packages,"  which  are  blocks  within 
Block  II.  This  plan  would  result  in  procurement  and  fielding  11  sets  of 
equipment  to  replace  the  11  operational  Block  I  sets  before  Milestone  III,  half 
of  the  22  total  sets  needed  by  the  Army  as  specified  in  the  ROC.  However,  as 
of  October  1992,  the  Program  Office  requir^  28  sets  for  Block  11.  Because  the 
Block  I  systems  would  be  incrementally  upgraded  throughout  Block  11 
development  and  fielded  to  the  11  units  comprising  Force  Package  1,  the 
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Block  I  and  Block  H  ASAS  would  be  fielded  without  the  Program  having  a 
form^  milestone  review  and  prior  to  a  DAB  Milestone  III  decision. 
Procurement  and  fielding  of  ASAS  systems  to  all  11  of  the  Army's  Force 
Package  1  units  is  especially  significant  since  the  ROC  states  that  only  22  ASAS 
systems  will  be  procured. 

In  a  May  20,  1992,  acquisition  policy  memorandum,  the  USD(A)  stated  that  the  - 
dramatic  lessening  in  the  threat  of  a  large-scale  war  in  Europe  permits  the 
Unit^  States  to  reduce  the  size  of  its  armed  forces  significantly.  As  the 
original  ASAS  requirement  was  based  on  a  larger  Army  and  a  large-scale 
European  war  scenario,  we  believe  that  a  more  realistic  requirement  is  closer  to 
11  systems  than  to  22,  despite  the  Program  Office's  current  estimate 
of  28  systems.  Thus,  with  11  Block  II  systems  fielded  by  1997,  Milestone  HI 
may  be  less  tiian  fully  effective  as  scheduled.  Further,  a  formal  Milestone  11 
review  is  in  order  because  of  the  possible  diminished  effectiveness  of  the  1997 
Milestone  HI  review. 

No  Low-Rate  Initial  Production.  No  LRIP  was  established  for 
Block  I,  although  since  October  1989  DOT&E  had  stated  in  Defense 
Acquisition  Executive  Summary  (DAES)  assessments  that  the  ASAS  (Block  I) 
was  in  LRIP  without  a  Milestone  IIIA  decision  or  an  operational  assessment. 
Further,  before  October  1992,  no  LRIP  was  required  for  Block  n  despite 
comments  by  DOT&E  in  May  and  July  1992  DAES  assessments  that  an  LRIP 
for  Block  n  must  be  established.  Rather,  the  Program  was  planned  to  proceed 
directly  to  full-rate  production.  The  United  States  Code,  title  10,  section  2400 
states  that  the  quantities  of  a  system  to  be  procured  for  LRIP  shall  be  established 
when  the  decision  is  made  to  approve  full-scale  engineering  development.  It 
further  defines  LRIP  for  a  new  system  as  the  minimum  quantity  necessary  to 
provide  production-configured  or  representative  articles  for  operational  tests, 
establish  an  initial  production  base,  and  permit  an  orderly  increase  in  the 
production  rate  sufficient  to  lead  to  full-rate  production  upon  successful 
completion  of  operational  testing. 

Decisions  that  resulted  in  the  Program  restructuring  and  redirection  effectively 
skirted  the  statutorily  imposed  requirements  concerning  limitations  on  LRIP 
quantities  and  need  for  operational  test  and  evaluation.  This  was  done  through 
the  Army's  use  of  an  accelerated  procurement  strategy  (i.e.,  "lihiited 
procurement-urgent")  in  1986  and  1988  and  incorporation  of  Hawkeye 
technology  in  1991,  thus  precluding  the  need  for  further  procurement  of  Block  I 
systems.  Since  all  Block  I  systems  will  be  replaced  during  EMD  by  Block  II 
systems,  the  Program  requires  a  formally  defined  LRIP  decision  point  based  on 
USD(A)-approv^  exit  criteria. 

Block  n  Will  Restart  Acquisition  ^asell.  Like  Block  I,  Block  II 
will  also  be  a  limited  capability  configuration  that  will  meet  only  four  of 
six  primary  ASAS  functions.  Specifically,  the  four  functions  met  by  Block  I 
and  II  are  intelligence  collection  management,  signjd  intelligence  management, 
situation  analysis,  and  target  analysis;  the  two  functions  not  met  are  electronic 
warfare  support  and  operations  security  support.  According  to  the  Block  II 
Master  Schedule  Replan  of  August  1992,  the  Block  II  ASAS  will  be  developed 
and  fielded  to  Force  Package  1  units  in  five  phases  from  FY  1993  through 
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FY  1996.  As  a  result,  significant  overlap  will  occur  with  fielding  of  Block  I 
units,  scheduled  for  fielding  from  FY  1993  through  FY  1995.  This  overlap 
n^s  to  be  addressed  through  review  of  both  Block  I  materiel  release  and  the 
overall  Block  II  program  to  determine  whether  fielding  of  Block  I  is  nec«ss^ 
concurrent  with  replacement  by  the  Block  n  system.  The  minimum  objective 
capability  will  not  be  met  until  Block  m,  scheduled  for  fielding  in  FY  2002. 

We  consider  Block  II  to  be  a  new  program  that  should  be  required  to  meet  all 
prerequisites  for  entry  into  EMD.  The  Block  n  contract  is  for  EMD  of  both 
new  hardware  and  software  compared  to  the  hardware  and  sof^are  used  in 
Block  I.  More  than  simply  remaining  in  acquisition  Phase  II  until  FY  1997  as 
implied  by  the  evolutionary  acquisition  strategy,  the  ASAS  Program  will  restart 
EMD  for  an  essentially  new  system  with  award  of  the  Block  11  contract.  M 
Block  I  equipment  will  be  replaced  by  common  hardware  and  Mftware  in 
Block  n,  with  the  hardware  representative  of  the  eventual  objective  system. 
Therefore,  Block  II  hardware  should  be  managed  through  an  effective  systems 
engineering  program  that  includes  design  and  production  readiness  reviews, 
structured  LRIP  to  confirm  manufacturing  processes  and  operational 
capabilities,  and  detailed  exit  criteria  for  proofing  both  within  EMD  and  to 
full-rate  production. 

The  Block  II  software  upgrade  consists  of  the  ASAS  Block  I  limited  capability 
system  repackaged  to  the  software  specifications  of  the  ATCCS.  Portions  of  the 
ASAS  software  will  be  converted  to  an  raitirely  new  language.  Additional 
software  will  be  developed  to  meet  the  objective  system  requirements  of  the 
four  Block  II  functions,  including  the  use  of  common  ATCCS  support  software 
and  an  open  system  architecture,  capability  for  continued  operations  in  a 
degraded  environment  or  condition,  incorporation  of  the  Collateral  Enclave,  and 
diagnostic  testing.  The  software  for  the  addition^  Block  HI  functions  that 
comprise  the  objective  system  will  be  developed  starting  in  FY  1997.  With  this 
plan,  required  design  reviews  and  testing  may  be  bypassed  by  an  evolutionary 
acquisition  strategy  that  does  not  comply  with  the  rigors  of  a  formal  EMD 
program.  Given  me  risk  associated  with  the  integration  of  the  overall  ATCCS 
software  architecture,  this  plan  should  not  be  implemented. 

In  summary,  we  consider  the  Block  II  program  to  require  a  formal  Milestone  n 
review  as  a  major  Defense  acquisition  program  entering  EMD.  The  planned 
evolutionary  acquisition  strategy  is  not  appropriate  for  the  ASAS  hardwme  and 
should  not  be  used  to  bypass  essential  aspects  of  a  properly  disciplined 
developmental  program  for  both  the  hardware  and  software.  Program  direction, 
including  an  approved  acquisition  program  baseline  for  development  and 
specific  exit  criteria,  is  essential  from  the  milestone  decision  authority  to  ensure 
the  evolutionary  acquisition  strategy  properly  provides  for  completion  of 
prerequisites  to  production  and  deployment. 

Blocks  IV  and  V  Excluded  From  Baseline.  As  of  October  1992,  the 
current  Acquisition  Strategy  Report,  ROC,  and  TEMP  identified  that  the  ASAS 
would  be  acquired  in  five  blocks.  The  Budget  Item  Justification  Sheet  dated 
January  1992  and  submitted  with  the  FY  1993  President's  Budget  also  described 
the  Program  as  consisting  of  five  blocks.  The  TEMP,  dated  October  1991, 
stated  that 
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The  AS  AS  Block  IV  andV  upgrades  will  improve  and 
enhance  existing  capabilities,  specifically  in  the  areas  of 
Intelligence  Collection  Management,  All  Source  Analysis, 

Signals  Intelligence  and  Situation  Development  applications 
in  FY  01-06.  Hardware  modifications  will  also  be  added  to 
allow  operations  on  the  move,  to  accoimt  for  obsolescence, 
and  to  take  advantage  of  improvements  in  computer 
technology,  where  applicable. 

The  Acquisition  Strategy  Report,  dated  November  1991,  stated  that  Blocks  IV 
and  V  will  be  implemented  as  Post  Deployment  Software  Support  efforts  from 
FY  2002  through  FY  2012  and  that  all  user  requirements  as  defined  in  the  ROC 
will  be  fiilly  satisfied  upon  completion  of  Block  V.  However,  the  current 
acquisition  program  baseline  dated  December  1991  includes  costs  for  only  three 
blocks  and  the  Budget  Item  Justification  Sheet  for  FYs  1994-95,  submitted  to 
OSD  in  October  1992,  describes  the  Program  as  consisting  of  only  three  blocks. 
In  the  June  1992  DAES,  the  Program  Manager  responded  to  OSD  comments  on 
the  number  of  blocks  and  stated  that 

The  approved  baseline  includes  all  development  blocks.  .  .  . 

ASAS  briefed  the  CAIG'  that  its  baseline  would  cost 
three  development  blocks  of  ASAS.  Proposed  Blocks  IV 
and  V  were  excluded  because  the  PEO  CCS  determined 
these  developments  were  too  far  in  the  future  (past  2002)  to 
adequately  define  and  cost,  and  were  considered  part  of  a 
planned  ATCCS  common  development. 

DoD  Instruction  5000.2  requires  that  values  for  acquisition  program  baseline 
parameters  reflect  the  cost  of  the  system  as  it  is  expected  to  be  produced  or 
field^.  We  do  not  consider  "evolutionary  acquisition"  to  mean  the  total  cost  of 
a  program  is  not  disclosed  when  the  baseline  is  established  or  the  budget  is 
prepared.  If  the  operational  requirement  will  not  be  satisfied  until  Block  V  is 
completed,  then  the  program  cost  should  include  all  efforts  through  Block  V  to 
meet  the  overall  requirement. 

ASAS  Is  Not  a  True  Evolutionary  Acquisition  Program.  The  ASAS 
Program  employs  the  evolutionary  acquisition  strategy  developed  fot  the 
acquisition  of  command  and  control  systems.  DoD  Instruction  5000.2  provides 
for  such  an  alternative  approach  "...  where  requirements  refinements  are 
anticipated  or  where  a  technology  risk  or  opportunity  discourages  immediate 
implementation  of  a  required  capability. " 

The  evolutionary  acquisition  approach  is  described  in  the  "Joint  Logistics 
Commanders'  Guidance  for  the  Use  of  an  Evolutionary  Acquisition  Strategy 
(EA)  in  Acquiring  Command  and  Control  (C2)  Systems,"  Defense  Systems 
Management  College,  March  1987.  The  Guidance  states  that  "An  underlying 
factor  in  evolutionary  acquisition  is  the  need  to  field  a  well-defined  core 
capability  quickly  in  response  to  a  validated  requirement." 
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The  guidance  also  states  that  evolutionary  acquisition  is  not  an  exemption  from 
disciplined  configuration  management  or  testing  and  is  not  p  approach  that 
provides  for  unconstrained  requirements  growth  and  an  unbridled  budget.  It 
further  states  that  blocks  should  be  treated  like  separate  acquisitions. 

The  ASAS  does  not  meet  the  criteria  for  an  evolutionary  acquisition  program. 
While  it  is  a  software  intensive  system  that  will  be  incrementally  upgraded  to 
achieve  the  requirements  of  the  objective  system,  the  Army  did  not  plan  to 
properly  manage  the  individual  blocxs  as  separate  acquisitions.  The  ASAS  also 
failed  the  test  for  fielding  a  well-defined  core  capability  quickly  in  response  to  a 
validated  requirement.  ASAS  will  have  been  in  development  for  10  years 
before  the  core  system,  Block  I,  is  fielded.  The  "baseline"  system  will  not  be 
fully  developed  until  the  end  of  Block  II,  and  the  obj^tive  system  will  not  be 
achieved  until  Block  III.  The  ASAS  development  will  spM  30  years  for  the 
eventual  five  blocks.  The  evolutionary  approach,  in  our  opinion,  is  being  used 
for  justifying  the  circumvention  of  testing  r^uirements,  the  continued 
avoidance  of  DAB  oversight,  and  the  delay  in  preparation  of  Program 
documentation  defining  system  requirements  and  funding  profiles. 

Impact  of  No  DAB  Milestone  Review.  By  1997,  the  ASAS  Program 
will  not  have  had  a  DAB  milestone  review  for  15  years,  and  two  interim  blocks 
will  have  been  essentially  procured  and  fielded  at  a  cost  of  $1.75  biUion^. 
Block  n  IOT«&E  and  Milestone  HI  DAB  review  will  be  meaningless  since  little 
will  be  left  to  test,  procure,  or  field.  The  ASAS  Program  will  have  effectively 
circumvented  the  DAB  process  through  "evolutionary  acquisition." 


Issues  and  Documentation  for  DAB  Program  Review 


On  February  3,  1992,  a  pkinning  meeting  was  held  for  the  scheduled  July  1992 
program  review.  The  C^I  working  group  decided  that  the  program  review 
should  be  in  January  1993,  as  the  Block  II  contract  planned  award  date  was 
November  1992.  Although  the  C^I  action  officer  prepared  a  memorandum, 
dated  February  6,  1992,  for  the  OSD  staff  to  coordinate  its  position  on  the 
program  review,  a  Committee  Memorandum  was  not  issued  for  8  months  (over 
7  months  late),  causing  a  hiatus  in  official  guidance  for  the  program  review. 
On  October  9,  1992,  after  the  working  group  held  two  additional  planning 
meetings,  the  Committee  Chair  issued  the  memorandum,  "Major  Issues 
Guidance  for  ASAS  DAB  Program  Review."  The  October  guidance  was  more 
comprehensive  than  that  contained  in  the  February  memorandum;  however,  the 
guidance  did  not  include  the  materiel  release  decision  for  Block  I  and  did  not 
ensure  preparation  of  aU  documentation  needed  for  a  thorough  review. 

Block  I  Material  Release.  The  DAB  program  review  planned  for 
March  1993  did  not  include  the  materiel  release  decision  for  Block  I.  The 
Army  Systems  Acquisition  Review  Council  is  scheduled  to  make  the  decision  on 
April  30,  1993.  Army  Regulation  700-142,  "Materiel  Release,  Fielding,  and 
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Transfer,"  April  27,  1988,  describes  three  types  of  materiel  release:  full  release, 
conditional  release,  and  training  release. 

A  full  release  is  authorized  when  the  materiel  has  been  tested 
and  evaluated  and  meets  all  established  requirements  of  the 
requirements  documents  and  specifications.  ...  A 
conditional  release  may  be  authenticated  when  one  or  more 
of  the  criteria  for  full  release  have  not  been  met.  .  .  . 

Materiel  release  for  training  may  include  prototype  or  test 
items,  .  .  .  items  that  are  incomplete  (major  components 
missing  or  defective),  and/or  items  where  one  or  more  of  the 
requirements  for  full  release  have  not  been  met. 

These  criteria  for  materiel  release  provide  wide  latitude  in  the  preconditions  for 
release  approval.  In  our  opinion,  because  of  the  inadequacies  of  the  testing 
program,  the  need  for  stringent  criteria  for  the  material  release  dedsion,  and  the 
high  risk  of  incurring  greater  costs  to  correct  problems  after  fielding,  the  DAB 
should  make  the  Block  I  materiel  release  decision. 

Documentation.  The  planned  DAB  program  review  does  not  ensure 
preparation  of  all  documentation  needed  for  a  thorough  review,  which  equates 
to  the  documentation  required  for  Milestone  II  (Appendix  A). 

The  C^I  action  officer  memorandum,  dated  February  6,  1992,  did  not  define  the 
critical  issues  to  be  addressed  and  documentation  required  for  the  January  1993 
program  review  as  required  by  DoD  Instruction  5000.2.  The  memorandum 
required  only  a  limited  cost  and  operational  effectiveness  analysis/exit  criteria, 
status  of  testing,  status  of  the  contract,  an  Integrated  Program  Summary  (less 
Annex  G),  a  Program  Life-Cycle  Cost  Estimate,  and  an  Independent  Cost 
Estimate.  Subsequently,  however,  the  C^I  Committee  Memorandum  of 
October  9,  1992,  required  the  Army  to  prepare  for  the  rescheduled  March  1993 
program  review  all  Milestone  H  documentation  not  previously  required  by  the 
C^I  action  officer  memorandum  of  February  6,  1992,  except  as  noted  below. 
Specifically,  the  Committee  Memorandum  additionally  required  the  Army  to 
prepare  an  ORD,  TEMP,  acquisition  program  baseline.  System  Threat 
Assessment  Report,  and  Manpower  Estimate  Report.  The  Committee  did  not 
require  an  Early  Operational  Assessment  Report,  needed  to  support  an  LRIP 
decision  with  exit  criteria  for  Block  II.  It  also  did  not  require  certain  key 
documentation  such  as  an  Independent  Cost  Estimate  Report,  a  Defense 
Intelligence  Agency  Intelligence  Report,  a  Joint  Requirements  Oversight 
Council  Assessment,  and  an  Integrated  Program  Assessment  to  be  prepar^  by 
OSD  and  other  Defense  organizations.  These  four  documents,  respectively, 
provide  independent  analysis  and  assessment  of  cost,  threat,  baseline 
performance  objectives  and  thresholds,  and  overall  program  status. 

We  contacted  the  Defense  Intelligence  Agency  and  the  OSD  offices  responsible 
for  preparing  the  four  documents.  They  generally  responded  that,  although  no 
requirement  exists,  the  documents  would  be  prepared  for  the  program  review. 
However,  because  preparation  and  review  of  all  needed  documentation  must  be 
ensured,  a  DAB  Milestone  review  is  warranted.  We  consider  the  actions  of  the 
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C^I  Committee  to  be  positive  and  believe  that  a  formal  Milestone  n  review  is 
prudent  since  Milestone  II  requirements  are  being  substantially  imposed  by  the 
Committee. 

Causes  for  Ineffectiveness  of  the  DAB  Process 


The  DAB  process  regarding  the  AS  AS  Program  was  not  effective  for  two  main 
reasons.  First,  the  C^I  Systems  Committee  did  not  begin  to  exert  any 
substantial  control  over  the  Program  until  almost  4  years  after  the  ASAS 
became  a  major  Defense  acquisition  program.  Then,  the  Army  continued  to 
control  the  Program  as  it  had  under  the  Joint  Oversight  Group  and  before  the 
ASAS  became  a  DAB  program. 

DAB  Control  Not  Exerted  for  4  Years.  Although  the  ASAS  became  a 
major  DAB  program  in  1987,  the  C^I  Systems  Committee  did  not  exert  any 
substantial  control  until  1991  when  some  basic  DAB  documentation  was  first 
required.  This  occurred  because  the  ASAS  ^ogram  ascribed  to  the 
evolutionary  acquisition  concept  and  the  C^I  Committee  had  allowed  ASAS  to 
deviate  from  conventional  requirements  for  major  Defense  programs.  Another 
factor  was  that  no  specific  criteria  existed  in  DoD  Instruction  5000.2  for 
evolutionary  acquisition  programs  and  no  guidance  existed  to  address 
documentation  requirements  for  an  ongoing  Defense  acquisition  program 
entering  DAB  program  status  between  major  milestones.  Also,  the  complexity 
of  the  Program,  frequent  programmatic  changes,  confusing  array  of  jargon  and 
acronyms,  and  turnover  of  ASAS  action  officers  within  C*!  have  made 
understanding  the  Program  difficult  and  contributed  to  the  weakness  in 
oversight. 

Anny  Control  Over  the  ASAS  Program.  In  the  absence  of  eff^tive 
DAB  oversight,  the  Army  had  continued  aggressively  to  control  the  direction  of 
the  ASAS  Program  as  the  Joint  Oversight  Group  had  done  prior  to  ASAS' 
becoming  a  DAB  program. 

In  November  1982,  the  operating  policy  of  the  ASAS  Program  was  established 
when  a  memorandum  from  the  Army  Chief  of  Staff  exempted  the  Program  from 
traditional  milestones  and  those  r^uirements,  except  legal,  which  "impede 
normal  systems  development  and  acquisition. " 

In  December  1991,  following  a  restructure  to  incorporate  Hawkeye,  the  ASAS 
Program  was  made  to  comply  with  some  DAB  documentation  requirements, 
including  a  baseline.  However,  the  baseline  reflated  only  three  blocks,  not  the 
full  five-block  Program  as  described  in  the  Acquisition  Strategy,  ROC,  TEMP, 
and  FY  1993  President's  Budget. 

In  June  1992,  following  a  funding  cut  in  the  Army  FYs  1994  through  1999 
Program  Objective  Memorandum,  the  ASAS  Program  was  again  restructur^. 
The  Army  revised  the  Block  11  plan  to  replay  two  equipment  enclaves  with 
one,  which  caused  changes  in  system  specifications  and  resulted  in  performance 
and  schedule  breaches  of  the  baseline.  Although  the  Program  Manager  bnefed 
OSD  and  submitted  a  Program  Deviation  Report  in  June  1992  with  a  new 
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baseline  attached,  no  OSD  approval  was  received.  However,  the  Program 
moved  ahead  as  if  OSD  approval  had  been  obtained.  For  instance,  the  Army 
amended  its  request  for  proposal  for  the  Block  II  contract  to  reflect  the 
restructure  before  a  new  baseline  was  submitted  to  OSD  for  approval. 

On  February?,  1992,  the  Army  Director  of  Information  Systems  for 
Command,  Control,  Communications,  and  Computers  sent  a  memorandum  to  - 
the  Deputy  Assistant  Secretary  of  Defense  (C^^  and  asked  for  his  help  to  ensure 
that  the  scope  of  the  DAB  review,  then  planned  for  January  1993,  remained 
limit^  to  award  of  the  Block  II  contract.  The  Director  stated,  "I  don't  feel  the 
standard  DoD  5000.2  'cookbook'  approach  applies." 

On  several  occasions  during  our  audit,  the  Program  Manager  and  other  Program 
officials  strongly  expressed  their  opinions  that  OSD  was  hindering  the  Program 
by  changing  documentation  requirements  and  withholding  funding.  For 
example,  they  cited  OSD  vacillation  about  whether  an  ORD  was  needed  instead 
of  the  ROC  and  whether  the  Collateral  Enclave  should  be  included  in  the 
TEMP.  Also,  they  did  not  believe  the  DoD  Comptroller,  a  non-signatory  to  the 
baseline,  should  have  the  capability  to  impede  execution  of  their  Program  by 
withholding  funds  and  adversely  affecting  Program  cost  and  schedule. 

On  August  26,  1992,  the  reply  by  the  Deputy  Under  Secretary  of  the  Army 
(Operations  Research)  to  a  DOT&E  memorandum  of  August  24,  1992, 
regarding  late  receipt  of  the  TEP  showed  the  Army's  intent  to  begin  lOT&E  on 
September  8,  1992,  regardless  of  DOT&E's  warning  that  the  TBP  may  not  be 
approved.  The  reply  stated,  "As  to  options  available  to  us  should  your  approval 
not  be  completed  prior  to  the  scheduled  start  of  test  date,  there  really  are  none 
that  are  reasonable.  ...  We  will  start  pilot  testing  on  8  September  with  record 
trials  scheduled  to  start  one  week  later." 


Conclusion 


We  disagree  that  the  overall  ASAS  Program  should  be  considered  to  havq  met 
Ae  requirements  for  entry  into  EMD  in  1983  based  solely  on  the  Block  I 
program.  The  ASAS  Program  redirection  by  the  USD(A)  in  1991  canceled 
production  of  all  but  a  limited  number  of  Block  I  systems  that  already  had  been 
produced  instead  of  proceeding  with  the  Block  II  program.  The  Army  has  not 
recognized  the  materiality  of  the  redirection  in  its  restructure  of  the  Program, 
relying  on  the  concept  of  "evolutionary  acquisition"  to  bypass  essential  program 
management  controls.  The  result,  in  our  opinion,  is  an  appearance  of  imminent 
success  to  a  program  ongoing  more  than  10  years,  when  the  system  has  not  been 
adequately  tested  and  does  not  meet  overall  user  requirements.  The  Block  II 
program  relies  on  substantially  different  hardware  and  software  than  Block  I  and 
is  planned  to  replace  the  Block  I  systems.  Additionally,  the  "objective  system" 
that  meets  the  minimum  user  requirements  will  not  be  attained  until  the 
Block  in  software  is  developed  and  implemented,  now  scheduled  for  FY  2002. 
The  Block  I  systems  are  neither  intended  to  meet  the  full  range  of  operational 


21 


Effectiveness  of  the  Defense  Acquisition  Board  Review  Process 


requirements  nor  reasonably  represent  the  production  hardware  and  software  to 
be  fielded  as  part  of  Block  n  or  subsequent  blocks. 

We  consider  the  ASAS  Block  I  program  to  equal  prototyping  done  during  the 
Demonstration  and  Validation  phase  of  the  acquisition  process  and  the  testing  as 
equiv^ent  to  an  operational  assessment  instead  of  the  more  rigorous  lOT&E.  In 
particular,  the  interoperability  of  Block  I  systems  with  the  overall  ATCCS 
program  has  not  been  thoroughly  tested,  alttiough  it  is  an  area  of  significant 
importance  and  potential  risk.  While  the  Army  has  identified  a  militaiy 
requirement  to  field  the  limited  capability  provided  by  the  Block  I  system,  this 
action  does  not  diminish  the  ne^  for  the  overall  ASAS  Program  to  meet 
requirements  for  a  DAB  milestone  review  prior  to  entry  into  Ehffi.  Although 
the  Program  has  not  been  required  to  pass  a  DAB  milestone  review,  the  Army 
plans  to  initiate  an  EMD  program  estimated  to  cost  $239  million  through 
FY  1997.  This  is  in  addition  to  procurement  costs  to  replace  Block  I  and  field 
Block  n  units,  with  future  Program  costs  through  Block  m  estimated  at 
$4.5  billion.  The  positive  actions  by  the  cognizant  DAB  Committee  to  conduct 
a  DAB  program  review  and  require  preparation  of  key  DAB  documents  do  not 
go  far  enough  to  ensure  adequate  program  oversight  in  a  DAB  Milestone  n 
review. 

Therefore,  we  believe  that  a  DAB  Milestone  II  review  of  the  ASAS  Program 
should  be  conducted  for  the  following  reasons: 

o  A  milestone  review  was  not  conducted  in  the  10  years  ASAS  has 
existed  and  was  not  planned  for  another  5  years. 

o  The  effectiveness  of  the  Milestone  III  review  scheduled  for  1997  may 
be  diminish^  because  of  the  plan  to  replace  all  11  operational  Block  I  systems 
with  Block  II  systems  by  1997. 

o  The  Block  I  materiel  release  decision  should  be  made  by  Ae  DAB 
because  of  the  inadequacies  of  the  testing  program,  the  minimal  criteria  for  an 
Army  release,  and  the  high  risk  of  incurring  greater  costs  to  correct  problems 
after  fielding. 

o  Block  II  would  effectively  restart  EMD  and  will  not  be  the  objective 

system. 


o  Blocks  IV  and  V  must  be  considered  part  of  the  ASAS  Program  and 
included  in  the  baseline  if  they  are  included  in  the  prospective  ORD  and 
derivative  documentation. 

o  All  documentation  needed  for  a  thorough  review  must  be  prepared. 
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Recommendations,  Management  Comments,  and  Audit 
Response 


1.  We  recommend  that  the  Under  Secretary  of  Defense  for  Acquisition 
conduct  a  Defense  Acquisition  Board  Milestone  11  review  of  the  All  Source . 
Analysis  System  Program  and  include  the  fielding  of  Block  I. 

2.  We  recommend  that  the  ComptroUer  of  the  Department  of  Defense 
withhold  all  remaining  FY 1993  Procurement  funds  of  $47.7  million 
appropriated  for  the  fielding  of  the  All  Source  Analysis  System  Block  I  and 
aU  remaining  Fyi993  Development  funds  of  $57.1  million  appropriated 
for  "Evolutionary  Acquisition"  of  Blocks  I  and  n  until  the  Under  Secretly 
of  Defense  for  Acquisition  issues  a  favorable  acquisition  decision 
memorandum  following  the  Defense  Acquisition  Board  Milestone  n  review 
for  the  All  Source  Analysis  System. 

Management  Comments.  We  received  comments  from  the  USD(A)  and  the 
DoD  ComptroUer.  The  USD(A)  also  included  comments  from  the  Army  for 
our  consideration.  The  USD(A)  partiaUy  concurred  with  Recommendations  1. 
and  2.  although  Recommendation  2.  was  not  addressed  to  the  USD(A).  The 
DoD  Comptroller  concurred  with  Recommendation  2.,  subject  to  concurrence 
by  the  USD(A).  Complete  comments  by  the  USD(A)  (including  the  Army 
comments)  and  the  DoD  Comptroller  are  in  Part  IV  of  this  report. 

Regarding  Recommendation  1.,  the  USD(A)  concurred  with  conducting  a  DAB 
Milestone  n  review  of  the  AS  AS  Program,  Also,  to  make  the  Block  II  EMD 
decision,  the  USD(A)  agreed  to  review  at  Milestone  H  the  results  of  Block  I 
lOT&E  and  the  Army's  plan  for  foUow-on  testing.  However,  the  USD(A) 
nonconcurred  with  including  the  fielding  of  Block  I  as  part  of  a  DAB  decision. 

Regarding  Recommendation  2.,  the  USD(A)  concurred  with  withholding  the 
ASAS  Block  II  funds  but  nonconcurred  with  withholding  Block  I  funds  because 
such  action  could  critically  hamper  ongoing  testing  and  preparations  for 
fielding. 

Regarding  Recommendation  2.,  the  DoD  Comptroller  stated  that  the  Block  II 
funds  were  being  withheld  pending  DAB  review  of  the  Program.  The 
Comptroller  also  proposed  withholding  Block  I  funds  subject  to  concurrence  by 
the  USD(A).  See  Appendix  B  for  the  DoD  ComptroUer’ s  response  to  the 
USD(A)  request  for  comments. 

Audit  Response.  Comments  by  the  USD(A)  are  not  fuUy  responsive. 

o  While  the  USD(A)  has  redesignated  the  upcoming  DAB  review  as  a 
formal  Milestone  H,  he  stUl  plans  for  the  Army  to  make  the  Block  I  fielding 
decision.  When  we  completed  field  work  in  December  1992,  the  Army 
Systems  Acquisition  Review  Council  was  scheduled  to  make  the  Block  I 
materiel  release  decision  in  April  1993,  after  the  DAB  review  then  planed  for 
March  1993.  This  sequence  of  events  would  have  met  the  intent  of 
Recommendation  1.  since  the  USD(A)  agreed  to  review  Block  I  lOT&E  and 
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plans  for  follow-on  testing  as  part  of  the  recommended  Milestone  11  review. 
However,  the  Army  now  plans  to  conduct  the  Block  I  materiel  release  decision 
before  the  DAB  review  due  to  the  scheduled  review  date  slipping  to  May  1993. 
Therefore,  we  ask  that  the  USD(A)  reconsider  his  position  and  either  rescind  the 
authority  for  the  Army  to  make  the  Block  I  materiel  release  decision  or  direct 
the  Army  to  defer  the  decision  until  after  the  DAB  review.  As  mentioned  in  the 
report,  the  Block  I  materiel  release  decision  should  be  considered  by  the  DAB 
because  of  inadequacies  of  the  testing  program,  the  minimal  criteria  for  an 
Army  release,  and  the  high  risk  of  incurring  greater  costs  to  correct  problems 
after  fielding.  Our  conclusion  is  further  supported  by  an  overall  operational 
evaluation  of  Block  I  by  the  Operational  Test  and  Evaluation  Command, 
December  21,  1992,  which  stated  that  the  ASAS  does  not  meet  operational 
effectiveness  criteria,  mission  performance  requirements,  suitability  criteria,  or 
sustainment  requirements. 

o  The  USD(A)  opposes  withholding  funds  for  the  fielding  of  Block  I. 
The  first  reason  given  is  that  the  USD(A)  delegated  Block  I  fielding  authority  to 
the  Army  and,  therefore,  the  associated  funds  should  remain  with  the  Army. 
We  reported  that  in  December  1991  the  USD(A)  did  not  plan  to  reconvene  the 
DAB  at  the  materiel  release  point  "unless  currently  unforseen  problems  arise." 
We  believe  that  recent  results  of  operational  testing  and  other  arguments  made 
in  oitf  report  regarding  Block  I  materiel  release  and  fielding  warrant 
reconsideration  of  the  earlier  USD(A)  decision.  The  other  reason  given  for 
opposing  the  withhold  is  that  it  could  critically  hamper  ongoing  testing  and 
preparations  for  the  fielding.  The  purpose  of  our  recommendation  is  ind^  to 
curtail  further  spending  on  Block  I,  particularly  regarding  fielding,  until  the 
DAB  reviews  the  Program.  Also,  as  we  have  stated  in  our  report,  the  testing 
program  is  invalid  and  inadequate  to  support  a  Block  I  fielding  decision. 
Therefore,  we  believe  that  no  further  funds  should  be  spent  on  testing  until  the 
DAB  reviews  the  Program. 

o  The  USD(A)  comments  do  not  indicate  concurrent  or 
nonconcurrence  with  the  material  internal  control  weaknesses  cited  in  our 
report. 

Concerning  the  USD(A)  comments  on  Recommendation  1.,  we  acknowledged 
in  our  report  that  the  purpose  of  the  DAB  program  review  was  to  examine  the 
Block  n  program  before  entry  into  EMD.  Further,  our  recommendation  was 
not  to  "entitle"  the  upcoming  DAB  review  a  Milestone  II  review  but  to  conduct 
one.  As  noted  in  our  report,  the  Committee  Memorandum  issued  in  October 
1992  required  most  of  the  Milestone  II  documentation  as  well  as  other  issues  to 
be  addressed  only  after  our  meetings  with  the  Deputy  Director,  Intelligence 
Programs  Support  Group,  identified  the  need  to  include  them  in  the  DAB 
review.  Moreover,  content  of  the  program  review  notwithstanding,  a 
Milestone  n  review  provides  greater  assurance  of  compliance  with  acquisition 
regulations. 

The  USD(A)  comments  contend  that  our  finding  on  the  effectiveness  of  the 
DAB  review  process  is  not  supported  by  the  audit  report  and  imply  the 
evjduation  is  not  credible  because  we  did  not  wait  until  the  DAB  review  was 
held  to  audit  and  evaluate  the  process.  Our  primary  purpose  in  auditing  and 
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reporting  before  the  DAB  review  is  to  provide  the  USD(A)  with  timely  and 
useful  information.  While  we  agree  that  to  delay  the  audit  would  provide  a 
more  complete  picture  of  the  process  for  historical  purposes,  the  result  would 
also  be  less  useful  for  decisionmaking.  The  DAB  review  is  the  end  product  of 
the  process  and  is  structured  by  that  process.  By  the  time  a  DAB  planning 
meeting  has  occurred  and  a  Committee  Memorandum  has  been  issu^,  all  major 
issues  and  required  documentation  should  be  identified.  At  this  point,  the  - 
framework  for  the  DAB  review  is  established.  We  believe  there  is  little  to  be 
gained  in  awaiting  the  outcome  of  the  DAB  review  and  only  then  to  report  on 
deficiencies  identified  when  available  corrective  actions  are  severely  limited. 

Comments  by  the  DoD  Comptroller  are  considered  responsive.  The 
Comptroller  communicated  his  comments  to  the  USD(A)  on  February  19,  1993, 
and  provided  us  with  a  copy  (Appendix  B).  In  particular,  he  noted: 

o  The  effectiveness  of  the  DAB  review  process  can  be  examined  on 
actions  taken  and  need  not  be  deferred  until  the  Milestone  n  review. 

o  The  DAB  review  planned  for  May  1993  may  not  occur  because  the 
Army  has  proposed  that  the  Block  II  program  be  deferred  in  the  FY  1994 
budget  adjustment  and  the  response  to  the  IG,  DoD,  should  reflect  this. 

o  The  adverse  reports  on  Block  I  testing  reveal  that  the  current 
configuration  is  not  effective  or  suitable  and  may  represent  problems  of 
sufficient  magnitude  to  reconvene  the  DAB  at  the  Block  I  materiel  release 
decision  point  in  accordance  with  the  USD(A)  memorandum  of  December  19, 
1991.  The  Comptroller  added  that  the  response  to  Recommendation  2.  should 
recognize  this  possibility. 
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Part  ni  -  Additional  Information 


Appendix  A.  Milestone  n  Documentation 

Requirements 

Operational  Requirements  Document 

System  Threat  Assessment  Report 

Defense  Intelligence  Agency  Intelligence  Report 

Joint  Requirements  Oversight  CouncU  Assessment 

Integrated  Program  Summary 

Integrated  Program  Assessment 

Program  Life-Cycle  Cost  Estimate 

Acquisition  Program  Baseline  Agreement 

Manpower  Estimate  Report 

Test  and  Evaluation  Master  Plan 

Developmental  Test  and  Evaluation  Report 

Independent  Cost  Estimate 

Independent  Cost  Estimate  Report 

Cost  and  Operational  Effectiveness  Analysis 

Early  Operational  Assessment  Report 
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the  Under  Secretary  of  Defense 
for  Acquisition 


OFFICE  OF  THE  COMPTROaER  OF  THE  DtPARTMEt^  OF  DEFENSE 
WASHINGTON.  DC  ZOSOIMIOO 


(Program/Budget)  FEB  1  9  1993 


MEMORANDUM  FOR  DEPUTY  DIRECTOR,  ACQUISITION  SYSTEMS  MANAGEMENT 

SUBJECT;  Draft  DoO  IG  Audit  Report  on  the  All  Source  Analysis 
System  (ASAS)  Program  (No.  2AE-0033.02) 


Your  memorandum  of  February  10,  1993  requested  our  review  of 
the  proposed  Under  Secretary  of  Defense  for  Acquisition  response 
to  the  subject  report.  We  do  not  concur  with  the  proposed 
response  for  the  following  reasons: 

1)  The  second  paragraph  suggests  that  the  audit  report  on 
the  effectiveness  of  the  Defense  Acquisition  Board  review 
process  is  premature  because  the  report  docs  not  address  the 
forthcoming  review  of  Block  II,  originally  planned  for  July  1992 
but  after  several  delays,  now  planned  for  May  1993.  There  was  a 
review  of  the  ASAS  program  by  the  Defense  Acquisition  Board 
principals  in  the  fall  of  1991  that  culminated  in  the  decision 
memorandum  signed  by  the  Under  Secretary  of  Defense  for 
Acquisition  on  December  19,  1991.  The  effectiveness  of  the 
Defense  Acquisition  Board  review  process  can  be  examined  on 
actions  taken  by  the  Defense  Acquisition  Board  to  date  and  need 
not  be  deferred  until  the  forthcoming  Milestone  II  review. 

2)  The  review  planned  for  May  1993  may  not  occur  because 
the  Army  has  proposed  that  the  Block  II  program  be  deferred  in 
the  FY  1994  budget  adjustment.  If  this  funding  reduction  is 
accepted,  there  will  be  little  need  for  a  Defense  Acquisition 
Board  review  of  Block  II  in  May  1993.  The  proposed  response 
should  be  modified  to  reflect  current  program  plans. 

3)  The  decision  memorandum  signed  by  the  Under  Secretary  of 
Defense  for  Acquisition  on  December  19,  1991  indicated  that  the 
Defense  Acquisition  Board  would  not  be  reconvened  at  the  Block  I 
material  redistribution  decision  point  unless  unforeseen 
problems  arose.  The  adverse  reports  regarding  the  results  of 
recent  Block  I  testing  in  the  January  1993  Defense  Acquisition 
Executive  Summary,  together  with  the  adverse  commentary  in  the 
Army  attachments  to  your  memorandum  that  indicate  that  testing 
reveals  that  the  current  configuration  is  not  effective  or 
suitable,  may  represent  unforeseen  problems  of  sufficient 
magnitude  to  reconvene  the  Defense  Acquisition  Board  at  the 
Block  I  material  redistribution  decision  point  in  accordance 
with  the  decision  memorandum  of  December  19,  1991.  The  proposed 
response  to  Recommendation  2  of  the  subject  report  should 
recognize  this  possibility.  If  sufficient  reason  to  reconvene 
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Appendix  C.  Summary  of  Potential  Benefits 

Resulting  From  Audit 

Recommendation  Amount  and/or 

Reference  Description  of  Benefit  Type  of  Benefit 


1. 


2. 


Program  Results.  Would  ensure  a  more  Nonmonetary. 

comprehensive  assessment  of  the  ASAS 

Program. 


Economy  and  Efficiency.  Would  ensure 
funds  are  spent  only  after  thorough 
review  and  approval  of  all  key  issues  by 
USD(A). 


Not  quantifiable 
because  the  cost  of 
correcting  Block  I 
deficiencies  after 
fielding  is  unknown. 
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Appendix  D.  Activities  Visited  or  Contacted 


Office  of  the  Secretary  of  Defense 


Under  Secretary  of  Defense  for  Acquisition,  Washington,  DC 
Director,  Defense  Research  and  Engineering,  Washington,  DC 
Deputy  Director,  Defense  System  Procurement  Strategies,  Washington,  DC 
Deputy  Director,  Test  and  Evaluation,  Washington,  DC 
Assistant  Secretary  of  Defense  (Command,  Control,  Communications  and 
Intelligence),  Washington,  DC  ,  .  v  ^ 

Assistant  Secret^  of  Defense  (Program  Analysis  and  Evaluation),  Washington,  DC 
Comptroller  of  the  Department  of  Defense,  Washington,  DC 
Director,  Operational  Test  and  Evaluation,  Washington,  DC 


Department  of  the  Army 


Office  of  the  Under  Secretary  of  the  Army  (Operations  Research),  Washington,  DC 
Assistant  Secretary  of  the  Army  (Financial  Management),  Washington,  DC 
Assistant  Secretary  of  the  Army  (Research,  Development  and  Acquisition), 
Washington,  DC 

Office  of  the  Army  Chief  of  Staff,  Washington,  DC 

Office  of  the  Deputy  Chief  of  Staff  for  Operations  and  Plans,  Washington,  DC 
Army  Materiel  Systems  Analysis  Activity,  Army  Materiel  Command,  Aberdeen,  MD 
All  Source  Analysis  System  Program  Office,  Army  Command  and  Control  Systems, 
McLean,  VA 

U.S.  Army  Cost  and  Economic  Analysis  Center,  Arlington,  VA 


Other  Defense  Organizations 

Defense  Intelligence  Agency,  Washington,  DC 
Joint  Staff,  Washington,  DC 
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Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition 

Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and  Intelligence) 
Assistant  Secretary  of  Defense  (Program  Analysis  and  Evaluation) 

Assistant  Secretary  of  Defense  (Public  Affairs) 

Comptroller  of  the  Department  of  Defense 


Department  of  the  Army 

Secretary  of  the  Army  ... 

Assistant  Secretary  of  the  Army  (Research,  Development  and  Acquisition) 
Inspector  General,  Department  of  the  Army 
Auditor  General,  Army  Audit  Agency 
Commanding  General,  Army  Materiel  Command 


Department  of  the  Navy 

Naval  Audit  Service  Headquarters 


Defense  Agencies 

Director,  Defense  Contract  Audit  Agency 

Inspector  General,  Defense  Intelligence  Agency 

Director,  Defense  Logistics  Agency 

Director,  Defense  Logistics  Studies  Information  Exchange 

Director,  National  Security  Agency 
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Non-DoD  Activities 

Office  of  Management  and  Budget  ... 

U.S.  General  Accounting  Office,  National  Security  and  International  Affairs  Division, 
Technical  Information  Center 

Chairman  and  Ranking  Minority  Member  of  the  Following  Congressional  Committees 
and  Subcommittees; 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
House  Committee  on  Government  Operations 

House  Subcommittee  on  Legislation  and  National  Security,  Committee  on 
Government  Operations 
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Under  Secretary  of  Defense  for 
Acquisition 

Comptroller,  Department  of  Defense 


Under  Secretary  of  Defense  for  Acquisition  Comments 


mm 


OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 

WASHINGTON,  DC  20301-3000 


February  24,  1993 

*‘'““'“t''memORANDUM  for  department  of  defense  inspector  general 

SUBJECT:  Draft  Audit  Report  on  the  All  Source  Analysis  System  ! 

(ASAS)  Program  As  a  Part  of  the  Audit  of  the  Defense  i 
Acquisition  Board  Review  Process  --  FY  1993 
(Project  No,  2AE-0033.02) 

We  have  reviewed  the  subject  draft  report  and  offer  the 
following  comments  concerning  recommendations  contained  in  the 
report.  Comments  from  the  Army  are  also  attached  to  this 
memorandum  for  your  consideration  in  preparing  the  final  report. 

The  stated  purpose  of  the  report  was  to  evaluate  the 
effectiveness  of  the  DAB  review  process  for  acquisition  of  major 
defense  acquisition  programs.  We  found  that  IG*s  finding,  that 
the  DAB  process  was  not  effective,  is  not  at  all  supported  by  the 
audit  report.  A  credible  evaluation  of  the  effectiveness  of  any 
process  would  require  the  process  to  have  been  completed.  We 
believe  that  a  judgment  on  DAB  effectiveness  in  reviewing  the 
ASAS  program  should  be  made  after  the  DAB,  now  scheduled  for  the 
end  of  May  1993,  is  held. 

The  following  comments  address  the  two  recommendations  in 
the  draft  report: 

po^«P>Tnn>pndation  #1>  That  USD  (A)  conduct  a  Milestone 

II  DAB  to  address  contract  award  for  Block  II  and 

fielding  of  Block  I, 

•  We  concur  with  your  recommendation  to  entitle  the 
upcoming  DAB  review  a  Milestone  II  rather  than  a 
program  review.  The  purpose  of  the  review  has 
always  been  to  review  the  readiness  of  Block  II 
for  the  Engineering  and  Manufacturing  Development 
(EMD)  phase.  The  major  issues  guidance  memorandum 
sent  to  the  Army  for  the  upcoming  DAB  has 
requested  all  of  the  documentation  required  for  a 
Milestone  II  DAB. 

•  With  regard  to  the  ASAS  Block  I  program,  USD (A) 
delegated  the  fielding  decision  to  the  Army,  and 
we  do  not  see  any  reason  to  reverse  that  decision 
at  this  time.  As  far  as  your  recommendation  to 
review  the  results  of  testing  of  Block  I  is 
concerned,  the  results  of  the  initial  operational 
test  and  evaluation  (lOT&E)  as  well  as  the  Army's 
plan  for  follow-on  testing  will  be  presented  to 
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the  DAB  as  a  part  of  the  information  required  for 
the  USD (A)  to  make  the  ASAS  Block  II  EMD  decision. 

Recommendation  #2.  That  the  DoD  Comptroller  withhold 
all  remaining  FY93  ASAS  Block  I  funds  ($47.7M)  and  all 
FY93  RDT&E  funds  ($57.1M)  for  the  ASAS  Block  I  &  II 
programs  pending  favorable  Milestone  II  DAB  decisions, 

•  We  support  the  withholding  of  the  ASAS  Block  II 
EMD  funds  until  the  DAB  review.  However,  we  do 
not  agree  with  the  recommendation  to  withhold  the 
ASAS  Block  I  funds  for  two  reasons:  First,  USD (A) 
made  an  acquisition  management  decision  on  the 
Block  I  program,  delegating  fielding  authority  to 
the  Army.  Therefore,  management  of  the  associated 
funds  required  for  Block  I  fielding  should  also 
remain  with  the  Army.  Secondly,  if  the  funds  were 
withheld,  it  could  critically  hamper  ongoing 
testing  and  preparations  for  the  fielding. 

We  believe  that  the  Army  should  continue  the  testing  of 
Block  I  for  fielding,  and  that  the  DAB  review  should  be  conducted 
as  originally  directed  by  USD(A) .  We  also  believe  that  to 
determine  the  effectiveness  of  the  DAB  process,  it  would  be  of 
more  value  if  DoD  IG  would  audit  and  evaluate  the  ASAS  DAB  review 
process  after  the  DAB  is  conducted. 


Gene  H.  Barter 
Principal  Deputy  Director 
Acquisition  Policy  and 
Program  Integration 
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DEPARIMENT  OF  IHt  ARMY 

OFFICE  OF  THE  SECRETARY  OF  THE  ARMY 
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OffiM.  Oirtctor  of  Information 
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MEMORANDUM  FOR  UNDER  SECRETARY  OF  DEFENSE  (ACQUISITION) 

SUBJECT:  Draft  DoD  IG  Audit  Report  on  the  All  Source  Analysis 

System  (ASAS)  Program  (Project  No.  2AE-0033.02) 


1.  References: 


a.  Draft  DoD  IG  Report,  28  Dec  92,  Subj;  Draft  Audit  Report 
on  the  Review  of  the  All  Source  Analysis  System  as  a  Part  of  the 
Audit  of  the  Effectiveness  of  the  Defense  Acquisition  Board 
Review  Process— FY  1993  (Project  No.  2AE-0033.02) 

b.  Memorandum,  Under  Secretary  of  Defense  (Acquisition), 

19  Dec  1991,  Subj:  ASAS  Program  Documentation. 

2,  The  Army  does  not  agree  with  the  two  recommendations 
contained  in  the  draft  DoD  IG  audit  report  (reference  a).  The 
first  recommendation  is  that  a  Milestone  II  DAB  review  of  the 
ASAS  program  be  conducted  to  include  the  fielding  of  ASAS  Block 
I.  We  believe  that  there  are  no  acquisition  issues  which  require 
a  DAB  review  of  the  ASAS  Block  I  materiel  release  decision,  and 
that  this  decision  should  remain  with  the  Army.  The  report  also 
recommends  that  all  remaining  FY  93  ASAS  funding  be  withheld 
pending  a  Milestone  II  DAB.  If  implemented,  this  recommendation 
would  stop  the  ASAS  Program  Manager's  efforts  to  prepare  for  a 
DAB  review.  Moreover,  the  findings  and  recommendations  are  not 
focused  on  the  DAB  review  process  which  is  ostensibly  the  primary 
objective  of  the  audit.  We  recommend  that  USD(A)  and  DoD 
Comptroller  not  concur  with  the  draft  Dod  IG  audit  report 
recommendations  and  that  the  report  be  withdrawn. 

3.  Milestone  II  DAB  Recommendation, 

a.  The  first  report  recommendation  is  that  the  USD(A)  should 
conduct  a  DAB  Milestone  II  review  of  the  ASAS  program  which 
includes  the  fielding  of  Block  I.  From  an  Army  perspective, 
upgrading  the  currently  scheduled  DAB  special  program  review  to  a 
formal  Milestone  II  DAB  is  a  difference  without  distinction  with 
regard  to  reviewing  the  ASAS  Block  II  Engineering  and 
Manufacturing  Development  (EMD)  contract  award.  The  Army  is 
already  complying  with  all  the  Milestone  II  DAB  documentation 
requirements.  The  Army  supports  the  USD(A)  position  expressed  in 
reference  b  that  ASAS  Block  I  materiel  release  is  an  Army 
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decision.  We  believe  that  there  are  no  acquisition  issues 
involved  in  fielding  the  on-hand  ASAS  Block  I  equipment  to  Army 
units  and  that  the  Army’s  materiel  release  decision  process  will 
ensure  that  the  equipment  is  operationally  effective  and  suitable 
before  it  enters  the  force. 

b.  The  draft  audit  report  cites  six  reasons  for  its  first 
recommendation.  Army  comments  on  these  reasons  follow: 

Reason:  ‘*A  milestone  review  was  not  conducted  in  the  10 
years  ASAS  has  existed  and  is  not  planned  for  another  5 
years . " 

Comment:  DAB  Milestone  reviews  are  event,  not  calendar 

driven.  An  event  requiring  a  DAB  milestone  review  has  not 
occurred  during  the  time  ASAS  has  been  a  DAB  program.  ASAS 
became  a  DAB  reviewed  program  in  1987  after  it  had  entered 
the  EMD  phase.  The  next  required  milestone  review  is  at  the 
completion  of  the  EMD  phase  in  FY  97. 

Reason:  “The  effectiveness  of  the  Milestone  III  review 
scheduled  for  1997  may  be  diminished  because  of  the  plan  to 
replace  all  11  operational  Block  I  systems  with  Block  II 
systems  by  then." 

Comment:  The  Array  has  no  plans  to  replace  Block  I  systems 

with  Block  II  systems  prior  to  a  DAB  Milestone  III 
production  and  deployment  decision  for  ASAS  Block  II. 

The  draft  report  incorrectly  asserts  that  as  of  Aug  92, 
the  Program  Manager  planned  to  field  Block  II  in  "capability 
packages,"  which  are  blocks  within  Block  II.  The  ASAS 
Project  Manager’s  "capability  package"  plan  is  to  improve 
Block  I  systems  after  they  are  fielded.  During  Block  II 
EMD,  the  contractor  will  be  required  to  provide  periodic 
deliverables  for  evaluation.  As  appropriate,  PM  ASAS  and 
CECOM  will  package  some  of  the  Block  II  advances  into 
capability  packages  that  can  be  applied  to  ASAS  Block  I. 

The  process  is  similar  to  how  M-60  and  M-48  main  battle 
tanks  were  continuously  improved  during  M-1  Abrams 
development.  The  Army’s  intent  is  to  keep  ASAS  Block  I 
technology  as  current  as  possible,  so  it  can  redistribute 
Block  I  systems  to  lower  priority  units  after  Block  II  is 
fielded. 

The  report  cites  the  plan  to  replace  Block  I  with  Block 
II  systems  prior  to  1997  several  times  (pp.  2,  19,  20,  and 
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21 ) .  We  recommend  that  findings  based  on  these  references 
be  reviewed  and  withdrawn. 

Reason:  "The  Block  I  materiel  release  decision  should  be 

made  by  the  DAB  because  of  the  inadequacies  of  the  testing 
program,  the  minimal  criteria  for  an  Army  release,  and  the 
high  risk  of  incurring  greater  costs  to  correct  problems 
after  fielding," 

Comment:  There  is  no  value  added  by  a  DAB  review  of  Block  I 

materiel  release.  The  testing  and  evaluation  program 
provides  adequate  information  for  the  Army  to  make  the 
materiel  release  decision.  The  Army  has  a  very  rigorous 
type  classification  and  materiel  release  approval  program 
which  ensures  that  equipment  is  operationally  effective  and 
suitable  before  it  is  fielded.  Finally,  Block  I  fielding 
will  reduce,  not  increase  ASAS  program  risk. 

The  draft  report  asserts  that  the  ASAS  test  and 
evaluation  program  is  inadequate.  The  U.S.  Army  Operational 
Test  and  Evaluation  Command  has  prepared  detailed  comments 
on  the  report  sections  addressing  the  adequacy  of  testing 
and  reviews  which  is  enclosed. 

The  IG  report  was  prepared  between  July  and  December 
1992,  but  it  does  not  discuss  the  conduct  of  the  ASAS  lOT&E; 
data  collected,  or  results.  Far  from  being  a  perfunctory 
"rubber  stamp,"  ASAS  testing  has  revealed  a  number  of 
hardware,  software  and  training  problems  which  need  to  be 
fixed  prior  to  fielding.  The  testing  also  revealed  a  number 
of  training  and  doctrine,  tactics,  techniques  and  procedures 
(DTT&P)  difficulties  that  the  Army  needs  to  address  as  the 
lEW  mission  area  is  automated.  Prior  to  a  conditional 
materiel  release  decision,  hardware  and  software  solutions 
will  be  evaluated  through  additional  operational  and 
technical  testing  during  March  and  early  April  1993.  A 
Follow-on  Operational  Test  and  Evaluation  will  be  conducted 
in  FY  94  before  an  unconditional  maberiel  release  decision 
is  made  by  the  Army,  The  ASAS  testing  and  evaluation 
program  is  clearly  providing  the  information  required  for 
decisions  on  Block  I  fielding. 

Regarding  risk  reduction,  the  Army  materiel  release 
decision  process  ensures  that  Block  I  deficiencies  are 
corrected  before  fielding  so  that  there  will  not  be  a  high 
risk  of  additional  costs  to  correct  problems  after  fielding. 
In  fact,  when  fielded  Block  I  will  reduce  the  training  and 
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doctrinal  risk  issues  associated  with  ASAS  development  since 
lessons  learned  from  field  experience  with  lEW  automation 
will  be  available  during  Block  II  development. 

Finally,  a  minor  argument  against  including  Block  I 
fielding  in  a  Milestone  II  DAB  is  that  the  purposes  stated 
in  DoDI  5000.2  for  a  Milestone  II  DAB  do  not  include  a 
materiel  release  decision  for  equipment  which  has  already 
been  procured.  The  audit  report  criticizes  lack  of  strict 
adherence  to  DoDI  5000.2  procedures  yet  makes  a  procedural 
recommendation  which  is  not  covered  by  the  acquisition 
directives . 

Reason:  "Block  II  would  effectively  restart  EMD  and  will 

not  be  the  objective  system." 

Comment:  Partially  agree.  As  correctly  stated  in  the 

report,  Block  III,  not  II,  is  the  objective  ASAS  system. 
However,  as  also  correctly  explained  in  the  report  (p.  22), 
the  Block  II  development  program  will  migrate  Block  I 
functionality  to  the  Army  Tactical  Command  and  Control 
System  (ATCCS)  common  hardware/software  open  systems 
architecture  and  then  improve  it.  Therefore,  Block  II  is 
not  a  complete  EMD  restart. 

Reason:  "Blocks  IV  and  V  must  be  considered  part  of  the 
ASAS  Program  and  included  in  the  baseline  if  they  are 
included  in  the  prospective  ORD  and  derivative 
documentation. " 

Comment:  The  ASAS  ORD,  which  was  approved  on  14  Jan  93, 

states  that  Block  III  is  the  objective  ASAS  system.  PM  ASAS 
is  required  to  build  the  objective  system.  Blocks  IV  and  V 
are  desired  enhancements,  not  baseline  requirements. 

Reason:  Preparation  of  all  documentation  needed  for  a 

thorough  review  must  be  assured." 

Comment:  In  preparation  for  the  currently  scheduled  DAB 

special  program  review,  the  Army  is  already  providing  all 
the  Army  produced  documents  required  for  a  Milestone  II  DAB, 

4.  Funding  Withhold  Recommendation.  The  second  draft  report 
recommendation  is  that  the  DoD  Comptroller  withhold  all  remaining 
ASAS  FY  93  funding  until  a  DAB  Milestone  II  is  completed. 
Withholding  remaining  FY  93  funding  would  effectively  shut  down 
the  ASAS  Project  Office.  PM  ASAS  would  not  be  able  to  prepare 
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for  the  DAB  review  discussed  in  the  report's  first 
recommendation.  The  DoD  Comptroller  Office  is  already 
withholding  $11.5  million  of  ASAS  Block  II  associated  funding 
pending  the  DAB  review.  We  believe  that  withholding  Block  I 
funding  until  a  DAB  review  is  unwarranted  since,  as  previously 
discussed.  Block  I  fielding  should  not  be  a  DAB  decision. 
Additional  withholds  would  critically  hamper  ongoing  testing 
efforts  and  DAB  and  fielding  preparations. 

5,  Additional  Areas  of  Concern.  The  Army  disagrees  with 
assertions  in  the  report  that  the  Army  used  Limited  Procurement- 
Urgent  procedures  and  an  evolutionary  acquisition  strategy  to 
circumvent  acquisition  oversight  in  the  ASAS  program.  In 
addition,  we  recommend  that  the  report  be  revised  to  include  more 
emphasis  on  the  DAB  review  process. 

a.  The  report  states,  "In  our  opinion,  the  Army  should  not 
have  used  expedited  "limited  procurement-urgent"  procedures  to 
bypass  essential  reviews  in  the  acquisition  process,"  (p.  17) 

Response:  The  Army  employed  valid  Limited 

Procurement-Urgent  procedures  to  procure  the  Limited  Capability 
Configuration-Europe  (LCC-E)  equipment  which  is  now  part  of  ASAS 
Block  I  in  an  attempt  to  meet  a  critical  operational  requirement, 
not  to  skirt  acquisition  oversight.  Since  1988  the  C3I  Committee 
has  reviewed  ASAS  seven  times  and  has  approved  significant 
changes  in  the  ASAS  program's  structure  and  schedule.  Prior  to 
then  the  Assistant  Secretary  of  Defense  for  Command,  Control, 
Communications  and  Intelligence  (ASD(C3l))  was  a  member  of  the 
Joint  Oversight  Group  which  provided  programmatic  oversight  for 
the  Joint  Tactical  Fusion  Program  which  managed  ASAS  development. 
Although  the  Central  Region  threat  has  disappeared,  the  urgency 
with  which  tactical  commanders  improvised  ASAS-like  intelligence 
automation  architectures  during  operations  DESERT  SHIELD/ STORM 
and  RESTORE  ROPE  reaffirm  the  urgency  of  the  requirement  and  the 
wisdom  of  the  LCC-E  decision. 

b.  The  report  also  states,  *‘The  ASAS  Program  will  have 
effectively  circumvented  the  DAB  process  through  'evolutionary 
acquisition,’"  (p.  25)  and  concludes  that  ASAS  is  not  a  true 
evolutionary  acquisition  program. 

Response:  The  report  conclusion  that  the  ASAS  Program 
circumvented  the  DAB  process  through  evolutionary  acquisition  is 
based  on  the  assertion  that  Block  II  systems  will  be  fielded 
before  a  Milestone  III  decision.  As  previously  discussed,  this 
assertion  is  incorrect.  The  report  concludes  that  ASAS  is  not  an 
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evolutionary  acquisition  program  because  a  core  capability  has 
not  been  rapidly  fielded  and  because  the  Army  "did  not  plan  to 
properly  manage  the  individual  blocks  as  separate  acquisitions.  ' 
(p.  24)  We  believe  that  the  ASAS  program  meets  the  evolutionary 
acquisition  program  criteria.  The  Army  is  attempting  to  field 
Block  I  which  will  provide  a  functional  baseline  upon  which  to 
build  subsequent  blocks^  and  Blocks  II  and  III  will  be  developed 
with  separate,  competitively  awarded  contracts. 

c.  The  draft  report  states  that  the  overall  objective  of 
the  audit  was  to  evaluate  the  DAB  review  process  of  the  ASAS 
system.  Yet  the  draft  audit  report  makes  no  recommendations  to 
correct  some  of  the  causes  the  report  associates  with  the  basic 
finding  that  the  DAB  review  process  was  not  effective  for  the 
ASAS  program.  Causes  cited  in  the  report  include  lack  of  DoDI 
5000.2  guidance  for  evolutionary  acquisition  programs  and  lack  of 
documentation  requirements  for  ongoing  programs  entering  DAB 
program  status  between  major  milestones  (p.27).  We  recommend 
that  the  audit  place  more  emphasis  on  DAB  review  process  issues 
such  as  these  and  include  the  major  issue  of  how  the  acquisition 
process  can  be  changed  to  accommodate  the  unique  problems 
associated  with  software  intensive  systems. 


SAFM-FO 
BALL 
SAPA-PP 
SAAG-PRP 
DACS-DM 
DAMO-FDI 
CDR,  OPTEC 
PEO  CCS 
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ALEXANDRIA*  VIRGINIA  22302  •  145S 


CSTE-OPM 


21  January  1993 


MEMORANDUM  FOR  Deputy  Under  Secretary  of  the  Army 

(Operations  Research),  ATTN:  SAUS-OR, 
Washington,  DC  20310 

SUBJECT;  Comment  on  the  DOD  IG  Draft  Audit  Report  on  the  Review 
of  ASAS  as  a  Part  of  the  Audit  of  the  Effectiveness  of  the 
Defense  Acquisition  Board  Review  Process  —  FY  93  (Project  No, 
2AE-0033.02) 


1.  Enclosed  is  proposed  response  to  subject  report  for  inclusion 
in  DISC4  response  to  the  Under  Secretary  of  Defense 
(Acquisition) .  It  addresses  those  sections  of  the  report 
pertinent  to  the  *' Adequacy  of  Testing  and  Reviews*'. 

2.  The  OPTEC  ODCSOPS  POC  is  Major  Steve  Reaves,  703-756^1516. 

FOR  THE  COMMANDER: 


DAVID  B.  KEN'^ 

LTC,  GS 

Deputy  Chief  of  Staff 
for  Operations 


Director,  Information  Systems  for  Command,  Control, 

Communications  and  Computers,  ATTN:  SAIS-2A,  Washington,  DC 
20310 

Commander,  TEXCOM,  Fort  Hood,  TX  76544-5065 

Commander,  OEC,  4501  Ford  Avenue,  Alexandria,  VA  22302-1458 


BUY  US  SAVINGS  BONDS  THROUGH  THE  PAYROLL  DEDUCTION  PLAN 


r/16/ 
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1.  Subject  report  has  been  reviewed.  The  following  comments, 
pertinent  to  those  sections  addressing  the  ‘'adequacy  of  Testing 
and  Reviews”,  are  provided  for  purposes  of  clarification  and 
accuracy.  Request  the  draft  report  be  amended  accordingly  prior 
to  publication. 

a.  The  report  advances  the  premise  that  testing  conducted 
and  testing  planned  for  ASAS  Block  I  are  inadequate  to  support  a 
decision  to  field  Block  I  equipment,  or  award  the  Block  II  EMD 
contract.  This  conclusion  is  based  upon  an  apparent 
misperception  as  to  the  purpose  of  operationally  testing  ASAS. 

The  test  program,  and  specifically  the  lOTE,  are  to  support  a 
materiel  release  decision,  which  the  IG  report  refers  to  as 
fielding,  and  not  the  Block  II  EMD  contract  award. 

b.  The  Block  I  ASAS  was  initially  intended  for  low  rate 
initial  production  for  the  objective  system.  Development 
problems,  in  concert  with  changes  to  the  threat  and  the  Army's 
role  in  national  security,  mandated  changes  to  the  ASAS  program. 
Block  I  is  therefore  now  intended  for  fielding  as  an  interim 
system;  no  further  production  decisions  on  Block  I  are  required. 

c.  The  draft  report  cites  selected  outcomes  of  the  1989 
Force  Development  Test  and  Experimentation  (FDTE)  as  evidence  of 
flawed  test  and  evaluation.  Particular  notice  is  taken  of  the 
failure  to  establish  a  performance  baseline  during  the  test  and 
of  the  continuation  with  further  testing  despite  equipment 
failures  during  Force  Development  Test  and  Evaluation  (FDTE) . 

The  report  implies  that  FDTE  should  have  been  repeated  because  of 
the  failure  of  a  message  parser. 

(1)  FDTE  is  conducted  principally  in  support  of  the 
Combat  Developer.  It  enables  him  to  refine  doctrine,  training 
and  logistics  concepts  for  a  particular  system.  It  is  also  a 
source  of  valuable  lessons  on  how  to  operationally  test  a 
system.  Test  and  evaluation  of  equipment  are  secondary  and,  most 
often,  the  Army  relies  upon  surrogate  equipment.  Failure  of  a 
piece  of  equipment  or  a  major  change  to  equipment  are  not  grounds 
to  repeat  a  doctrinal  experiment, 

(2)  The  absence  of  a  performance  baseline  does  not 
obviate  the  validity  of  future  operational  testing  and 
evaluation.  Systems  are  tested  and  evaluated  against  stated 
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»»User  Requirements”,  Baselines  are  useful  in  assessing  system 
maturity,  e,g.,  the  applicable  reliability,  availability,  and 
maintainability  (RAM)  data  collected  for  the  Portable  ASAS 
Workstation  and  the  Forward  Sensor  Interface  and  Control  during 
FDTE  were  data  sources  for  the  Initial  Operational  Test  and 
Evaluation  (lOTE)  of  these  functions.  However,  baselines  are  not 
the  measure  for  assessing  effectiveness  and  suitability. 

Further,  there  has  never  been  a  requirement  to  establish  an  ASAS 
operational  baseline. 

d.  The  draft  report  questions  the  validity  of  the  lOTE  on 
the  basis  that  it  did  not  address  interoperability,  the 
representativeness  of  the  system  tested,  and  scheduling  lOTE 
after  the  production  of  most  ASAS  Block  I  units.  It  also  raises 
issues  with  the  duration  of  lOTE,  testing  of  only  one  unit 
configuration,  gathering  test  data  in  a  garrison  environment, 
whether  the  test  supports  a  fielding  decision  and  the  award  for 
the  Block  II  Engineering  and  Manufacturing  Development  contract. 
The  report  correctly  notes  that  the  Test  and  Evaluation  Plan 
(TEP)  for  the  lOTE  was  not  timely,  however,  the  contention  that 
the  TEP  did  not  provide  for  the  determination  of  operational 
effectiveness  and  suitability  is  incorrect. 

(1)  The  ASAS  was  the  first  Army  Tactical  Command  and 
control  System  (ATCCS)  Battlefield  Functional  Area  (BFA)  to 
undergo  lOTE,  No  other  ATCCS  BFA  was  at  a  sufficient  level  of 
maturity  to  permit  an  automated,  interoperable  system  to  interact 
in  the  electronic  mode.  Consequently,  interoperability  testing 
was  not  an  option  during  the  ASAS  lOTE. 

(2)  The  Collateral  Enclave  (CE)  did  not  undergo 
interoperability  testing  during  the  ATCCS  Early  User  Test  and 
Experimentation  (EUTE)  in  October  1992;  the  technical  evaluator 
concluded  the  system  was  not  ready.  Interoperability  was 
successfully  demonstrated  during  the  command  post  exercise  phase; 
formal  testing  will  occur,  in  conjunction  with  ATCCS  testing  when 
the  CE  is  sufficiently  mature, 

(3)  The  auditors  conclusion  that  interoperability 
testing  was  not  included  in  the  TEP  is  incorrect.  The  lOTE  was 
designed  to  make  ASAS  receive  and  process  the  types  of  messages 
it  will  receive  from  national-level  assets.  Testing  Joint  and 
Allied  interoperability  was  planned  for  the  lOTE,  but  postponed 
to  permit  updating  of  message  formats. 

(4)  The  equipment  tested  during  lOTE  was  representative 
of  the  ASAS  Block  I  equipment.  The  assertion  contained  in  the 
report  to  the  contrary  is  false  and  misleading.  The  report 
correctly  states  that  most  of  the  Block  I  systems  had  been 
produced  prior  to  the  lOTE. 
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(5)  The  timing  of  the  lOTE,  with  respect  to  equipment 
production  does  not  invalidate  the  test;  the  purpose  of  the  test 
was  to  evaluate  operational  effectiveness  and  suitability.  The 
express  purpose  of  the  lOTE  was  to  support  a  Block  I  fielding 
decision  by  the  Army,  not  by  the  DAB  as  indicated  in  the  draft 
report.  The  statement  that  lOTE  did  not  support  a  production 
decision  is  correct.  A  production  decision  was  never  predicated 
on  this  lOTE. 

(6)  The  length  of  the  lOTE,  as  planned  and  executed, 
differs  from  the  length  stated  in  the  ASAS  TEMP  because  the 
necessary  data  could  be  collected  in  a  shorter  time  frame. 
Schedules  contained  in  the  TEMP  represent  planning  estimates; 
subsequent  deviations  do  not  invalidate  the  results.  On  the 
contrary,  it  would  be  irresponsible  to  prolong  testing  at 
increased  cost  to  satisfy  calendar  requirements  after  the 
necessary  data  to  satisfy  evaluation  objectives  had  been 
obtained. 

(7)  The  draft  report  cites  heavy  division  configuration 
used  during  the  test  as  a  limitation.  This  configuration  was 
selected  because  of  its  similarity  to  the  Corps  configuration.  It 
includes  all  equipment  found  in  the  Light  Division.  Testing 
additional  configurations  would  have  resulted  in  substantially 
higher  costs  with  little  return  on  the  investment. 

(8)  The  report  correctly  notes  that  much  of  the  data 
gathered  to  support  lOTE  was  gathered  in  garrison  vice  field 
operations.  This  is  not  a  test  limitation  as  the  auditors  imply. 
The  structure  of  the  lOTE  followed  the  long-established 
operational  pattern  of  military  intelligence  units  preparing  for 
and  executing  a  combat  mission.  A  unit  equipped  with  the  ASAS 
will  perform  many  operational  tasks  and  prepare  for  deployment 
while  in  garrison.  The  lOTE  captured  operational  data  on  ASAS 
performance  in  both  garrison  and  field  environments. 

(9)  Use  of  the  term  "opinion  poll"  to  describe  some  of 
the  methods  employed  for  determining  operational  effectiveness 
and  suitability  is  inappropriate  and  pejorative  as  it  is  applied 
to  the  accepted  practice  of  drawing  on  the  professional  military 
judgment  of  a  panel  of  subject  matter  experts  for  qualitative 
information.  The  implied  criticism  that  no  other  data  was  used  to 
assess  operational  performance  is  not  correct.  The  lOTE 
evaluation  makes  extensive  use  of  quantified  data. 

e.  The  draft  report  includes  the  incorrect  assertion  that 
the  ASAS  testing  program  is  invalid.  The  IG*s  evidence  to 
support  this  claim  is  drawn  from  the  FDTE,  the  integration 
between  the  HAWKEYE  and  CE,  and  the  differences  between  the  TEMP 
and  the  TEP. 
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(1)  The  HAWKEYE  underwent  testing  during  the  lOTE.  It 
will  be  evaluated  as  part  of  ASAS.  The  CE  is  not  part  of  Block  I 
ASAS,  neither  is  it  part  of  the  current  Block  I  fielding  decision 
because  the  enclave  has  not  been  operationally  tested  to  support 
a  materiel  release  decision.  The  CE  will  be  scheduled  for 
operational  testing  in  a  future  ATCCS  test  window.  If 
operationally  effective  and  suitable,  the  CE  is  likely  to  be 
fielded  with  Block  I  ASAS. 

(2)  Discrepancies  between  the  TEMP  and  the  TEP  have 
been  addressed  with  the  exception  of  the  change  of  critical 
quantitative  thresholds  to  non-critical  criteria.  This 
distinction  is  irrelevant  because  all  criteria  and  measures  of 
performance  were  considered  in  the  evaluation,  regardless  of  the 
criticality  label  applied.  In  short,  OPTEC  evaluates  operational 
effectiveness  and  suitability  and  is  not  limited  to  issues  and 
criteria  labeled  »’critical”. 

f.  The  auditors*  conclusion  that  testing  planned  for  Block  I 
lOTE  more  closely  resembles  an  operational  assessment  than  an 
operational  test  and  evaluation  appears  to  be  based  on  the 
misconceptions  discussed  earlier.  The  report  frequently  notes 
that  most  Block  I  ASAS  equipment  has  been  procured.  This  is 
irrelevant  to  the  determination  of  effectiveness  and  suitability 
by  us,  as  required  by  Congress  and  OSD.  The  ASAS  lOTE  met  the 
requirement  for  an  operational  test  and  evaluation;  an  approved 
threat,  typical  users,  and  production  representative  ASAS 
equipment,  were  included  in  the  lOTE. 


2.  It  is  unfortunate  that  the  operational  testers  and  evaluators 
were  not  consulted  during  the  course  of  this  audit;  a 
significantly  different  report  may  have  resulted.  The  ASAS  lOTE 
conducted  7  Sep  -  11  Oct  92  concluded  that  the  current 
configuration  of  the  system  is  not  effective  or  suitable.  The 
report  does  not  support  full  materiel  release  until  hardware, 
software,  procedural  and  training  problems  identified  during  lOTE 
have  been  corrected  and  verified  in  a  follow-on  test  and 
evaluation  program. 

3.  We  disagree  with  the  conclusion  of  the  draft  report  dealing 
with  the  validity  of  operational  testing.  They  are  based  on 
inaccurate  information  and  limited  understanding  of  the 
operational  testing  and  evaluation  process.  The  operational 
test  and  evaluation  program  for  ASAS  Block  I  was  valid  and 
adequate  to  support  required  decisions. 
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COMPTROLLER  OF  THE  DEPARTMENT  OF  DEFENSE 
WASHINGTON.  DC  20301-1  100 


JkN  22  G93 


MEMORANDUM  FOR  INSPECTOR  GENERAL,  DEPARTMENT  OF  DEFENSE 

SUBJECT:  Draft  Audit  Report  on  the  Review  of  the  All  Source 
Analysis  System  as  a  Part  of  the  Audit  of  the 
Effectiveness  of  the  Defense  Acquisition  Board  Review 
Process— PY  1993  (Project  No.  2AE-0033.02) 


Your  memorandum  of  December  28,  1992,  provided  a  copy  of  the 
subject  audit  report  that  recommended  funding  withholds  of  All 
Source  Analysis  System  development  and  procurement  funds. 

Funds  appropriated  in  FY  1993  for  Block  II  development  have 
been  on  withhold  since  the  beginning  of  the  fiscal  year.  These 
funds  will  remain  on  withhold  pending  Defense  Acquisition  Board 
review  of  this  program. 

Also,  in  response  to  the  report  recommendation,  remaining 
unobligated  and  uncommitted  FY  1993  development  and  procurement 
funds  for  Block  I,  except  for  program  office  costs  such  as 
salaries,  are  being  proposed  for  withhold  subject  to  concurrence 
by  the  Under  Secretary  of  Defense  for  Acquisition.  The  previously 
released  funds  that  have  now  been  obligated  and  committed  will 
continue  program  efforts  until  this  matter  is  resolved.  The  funds 
are  sufficient  to  allow  the  program  office  to  prepare  for  the 
forthcoming  Defense  Acquisition  Board  review. 

Donald  B.  ^ycoxr 
Acting  Comptroller 

cc:  USD(A) 
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Audit  Team  Members 


Donald  E.  Reed 

Russell  A.  Rau 
Brian  M.  Flynn 
Dennis  E.  Coldren 
Haskell  I.  Lynn 
Mary  Ann  Hourcld 
Tammy  O'Deay 


Director,  Acquisition  Management 
Directorate 
Program  Director 
Project  Manager 
Team  Leader 
Auditor 
Editor 

Administrative  Support 
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